2024-09-10T03:22:21.536 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-10T03:22:21.560 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T03:22:21.647 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-09-10_02:59:23-orch:cephadm-wip-adk-testing-2024-09-09-1904-distro-default-smithi/7898573 branch: wip-adk-testing-2024-09-09-1904 description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: adking@redhat.com first_in_suite: false job_id: '7898573' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/squid ' name: adking-2024-09-10_02:59:23-orch:cephadm-wip-adk-testing-2024-09-09-1904-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-adk-testing-2024-09-09-1904 ceph: cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: global: mon pg warn min per osd: 0 mgr: client mount timeout: 30 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - fs.*is degraded - filesystem is degraded - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - filesystem is offline - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - insufficient standby MDS daemons available - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy sha1: 3a5e3f16720f432aea965c40a8f6eb480095ec6c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3a5e3f16720f432aea965c40a8f6eb480095ec6c kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-adk-testing-2024-09-09-1904 sha1: 3a5e3f16720f432aea965c40a8f6eb480095ec6c owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 6964 sha1: 3a5e3f16720f432aea965c40a8f6eb480095ec6c sleep_before_teardown: 0 subset: 1/20 suite: orch:cephadm suite_branch: wip-adk-testing-2024-09-09-1904 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3a5e3f16720f432aea965c40a8f6eb480095ec6c targets: smithi112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDh2A+WkH8kC62PSSDud5mjSzbvJRsnvcVXII+NkyFp0gOxURyAPKFDX3E9jYTdkGbgJEADKJGffCzqmIbEybUQ= smithi169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFw7wZ/XCWkula3y0Bb6Ke5ibrgsxnwsPHDYOZv/YrUaESVj2XnF9ORXYeyi3QZNF0yg5DGTINHZ+ArbtnVYjpw= tasks: - install: branch: squid exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: compiled_cephadm_branch: squid conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:squid roleless: true - print: '**** done end installing squid cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_02:59:23 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-09-10T03:22:21.647 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa; will attempt to use it 2024-09-10T03:22:21.648 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks 2024-09-10T03:22:21.648 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-10T03:22:21.649 INFO:teuthology.task.internal:Checking packages... 2024-09-10T03:22:21.669 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3a5e3f16720f432aea965c40a8f6eb480095ec6c' 2024-09-10T03:22:21.669 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T03:22:21.669 INFO:teuthology.packaging:ref: None 2024-09-10T03:22:21.669 INFO:teuthology.packaging:tag: None 2024-09-10T03:22:21.669 INFO:teuthology.packaging:branch: wip-adk-testing-2024-09-09-1904 2024-09-10T03:22:21.669 INFO:teuthology.packaging:sha1: 3a5e3f16720f432aea965c40a8f6eb480095ec6c 2024-09-10T03:22:21.669 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-09-09-1904 2024-09-10T03:22:21.809 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4885.g3a5e3f16 2024-09-10T03:22:21.810 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-10T03:22:21.817 INFO:teuthology.task.internal:no buildpackages task found 2024-09-10T03:22:21.817 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-10T03:22:21.824 INFO:teuthology.task.internal:Saving configuration 2024-09-10T03:22:21.846 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-10T03:22:21.853 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-10T03:22:21.879 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-09-10_02:59:23-orch:cephadm-wip-adk-testing-2024-09-09-1904-distro-default-smithi/7898573', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 03:17:08.899200', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDh2A+WkH8kC62PSSDud5mjSzbvJRsnvcVXII+NkyFp0gOxURyAPKFDX3E9jYTdkGbgJEADKJGffCzqmIbEybUQ='} 2024-09-10T03:22:21.897 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-09-10_02:59:23-orch:cephadm-wip-adk-testing-2024-09-09-1904-distro-default-smithi/7898573', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 03:17:08.898258', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFw7wZ/XCWkula3y0Bb6Ke5ibrgsxnwsPHDYOZv/YrUaESVj2XnF9ORXYeyi3QZNF0yg5DGTINHZ+ArbtnVYjpw='} 2024-09-10T03:22:21.897 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-10T03:22:21.903 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-09-10T03:22:21.904 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-09-10T03:22:21.904 INFO:teuthology.run_tasks:Running task console_log... 2024-09-10T03:22:21.963 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7faa09073be0>, signals=[15]) 2024-09-10T03:22:21.963 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-10T03:22:21.971 INFO:teuthology.task.internal:Opening connections... 2024-09-10T03:22:21.971 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2024-09-10T03:22:21.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T03:22:22.047 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2024-09-10T03:22:22.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T03:22:22.119 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-10T03:22:22.125 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2024-09-10T03:22:22.144 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2024-09-10T03:22:22.144 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2024-09-10T03:22:22.202 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="CentOS Stream" 2024-09-10T03:22:22.202 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="9" 2024-09-10T03:22:22.202 INFO:teuthology.orchestra.run.smithi112.stdout:ID="centos" 2024-09-10T03:22:22.202 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE="rhel fedora" 2024-09-10T03:22:22.202 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="9" 2024-09-10T03:22:22.202 INFO:teuthology.orchestra.run.smithi112.stdout:PLATFORM_ID="platform:el9" 2024-09-10T03:22:22.202 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T03:22:22.202 INFO:teuthology.orchestra.run.smithi112.stdout:ANSI_COLOR="0;31" 2024-09-10T03:22:22.202 INFO:teuthology.orchestra.run.smithi112.stdout:LOGO="fedora-logo-icon" 2024-09-10T03:22:22.203 INFO:teuthology.orchestra.run.smithi112.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T03:22:22.203 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://centos.org/" 2024-09-10T03:22:22.203 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T03:22:22.203 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T03:22:22.203 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T03:22:22.203 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2024-09-10T03:22:22.226 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2024-09-10T03:22:22.244 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2024-09-10T03:22:22.244 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2024-09-10T03:22:22.302 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Stream" 2024-09-10T03:22:22.302 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="9" 2024-09-10T03:22:22.302 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos" 2024-09-10T03:22:22.302 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora" 2024-09-10T03:22:22.302 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="9" 2024-09-10T03:22:22.302 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el9" 2024-09-10T03:22:22.302 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T03:22:22.302 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2024-09-10T03:22:22.303 INFO:teuthology.orchestra.run.smithi169.stdout:LOGO="fedora-logo-icon" 2024-09-10T03:22:22.303 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T03:22:22.303 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/" 2024-09-10T03:22:22.303 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T03:22:22.303 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T03:22:22.303 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T03:22:22.303 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2024-09-10T03:22:22.325 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-10T03:22:22.334 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-10T03:22:22.346 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-10T03:22:22.346 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2024-09-10T03:22:22.348 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2024-09-10T03:22:22.366 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-10T03:22:22.371 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-10T03:22:22.372 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2024-09-10T03:22:22.405 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2024-09-10T03:22:22.507 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-10T03:22:22.550 INFO:teuthology.run_tasks:Running task kernel... 2024-09-10T03:22:22.565 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-10T03:22:22.565 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-10T03:22:22.565 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T03:22:22.566 DEBUG:teuthology.orchestra.run.smithi169:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T03:22:22.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T03:22:22.583 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-09-10T03:22:22.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T03:22:22.584 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-09-10T03:22:22.640 INFO:teuthology.orchestra.run.smithi112.stdout:5.14.0-503.el9.x86_64 2024-09-10T03:22:22.640 INFO:teuthology.task.kernel:Running kernel on smithi112: 5.14.0-503.el9.x86_64 2024-09-10T03:22:22.641 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum install -y kernel 2024-09-10T03:22:22.642 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-503.el9.x86_64 2024-09-10T03:22:22.642 INFO:teuthology.task.kernel:Running kernel on smithi169: 5.14.0-503.el9.x86_64 2024-09-10T03:22:22.642 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum install -y kernel 2024-09-10T03:22:24.997 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-09-10T03:22:26.246 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.2 MB 00:01 2024-09-10T03:22:29.297 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - AppStream 6.4 MB/s | 20 MB 00:03 2024-09-10T03:22:35.606 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.5 MB 00:02 2024-09-10T03:22:37.107 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - Extras packages 122 kB/s | 19 kB 00:00 2024-09-10T03:22:43.489 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 3.6 MB/s | 23 MB 00:06 2024-09-10T03:22:44.545 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 20 MB 00:17 2024-09-10T03:22:48.808 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-10T03:22:50.054 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-10T03:22:50.873 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T03:22:50.873 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T03:22:50.976 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:22:50.984 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:22:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-09-10T03:22:50.985 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:22:50.985 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-10T03:22:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T03:22:50.985 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-10T03:22:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T03:22:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T03:22:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T03:22:50.985 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:22:50.985 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-10T03:22:50.985 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:22:50.985 INFO:teuthology.orchestra.run.smithi112.stdout:Install 4 Packages 2024-09-10T03:22:50.986 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:22:50.986 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 79 M 2024-09-10T03:22:50.986 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 126 M 2024-09-10T03:22:50.986 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-10T03:22:51.093 INFO:teuthology.orchestra.run.smithi112.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 265 kB/s | 23 kB 00:00 2024-09-10T03:22:51.443 INFO:teuthology.orchestra.run.smithi112.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-09-10T03:22:51.571 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 117 kB/s | 19 kB 00:00 2024-09-10T03:22:51.835 INFO:teuthology.orchestra.run.smithi112.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 42 MB/s | 35 MB 00:00 2024-09-10T03:22:52.202 INFO:teuthology.orchestra.run.smithi112.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 26 MB/s | 29 MB 00:01 2024-09-10T03:22:52.202 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:22:52.203 INFO:teuthology.orchestra.run.smithi112.stdout:Total 65 MB/s | 79 MB 00:01 2024-09-10T03:22:52.853 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-10T03:22:52.884 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-10T03:22:53.253 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-10T03:22:53.253 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-10T03:22:54.187 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-10T03:22:54.187 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-10T03:22:56.656 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-10T03:22:57.352 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T03:22:57.436 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T03:22:58.239 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-10T03:23:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T03:23:00.272 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T03:23:00.272 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T03:23:00.357 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T03:23:00.368 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:23:00.373 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:23:00.373 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-09-10T03:23:00.374 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:23:00.374 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-10T03:23:00.374 INFO:teuthology.orchestra.run.smithi169.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T03:23:00.374 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-10T03:23:00.374 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T03:23:00.374 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T03:23:00.374 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T03:23:00.374 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:23:00.374 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-10T03:23:00.374 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:23:00.374 INFO:teuthology.orchestra.run.smithi169.stdout:Install 4 Packages 2024-09-10T03:23:00.374 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:23:00.375 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 79 M 2024-09-10T03:23:00.375 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 126 M 2024-09-10T03:23:00.375 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-10T03:23:00.691 INFO:teuthology.orchestra.run.smithi169.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 85 kB/s | 23 kB 00:00 2024-09-10T03:23:05.203 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T03:23:05.241 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T03:23:06.957 INFO:teuthology.orchestra.run.smithi169.stdout:(2/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 5.3 MB/s | 35 MB 00:06 2024-09-10T03:23:10.069 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T03:23:16.748 INFO:teuthology.orchestra.run.smithi169.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 981 kB/s | 16 MB 00:16 2024-09-10T03:23:31.905 INFO:teuthology.orchestra.run.smithi169.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 937 kB/s | 29 MB 00:31 2024-09-10T03:23:31.905 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:23:31.905 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.5 MB/s | 79 MB 00:31 2024-09-10T03:23:32.590 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-10T03:23:32.949 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-10T03:23:32.949 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-10T03:23:33.934 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-10T03:23:33.934 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-10T03:23:36.381 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-10T03:23:37.141 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T03:23:37.224 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T03:23:40.021 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T03:23:40.153 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T03:23:44.965 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T03:23:45.078 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T03:23:48.303 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T03:23:48.343 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T03:23:48.704 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T03:23:48.704 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T03:23:48.704 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T03:23:48.704 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T03:23:49.525 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T03:23:49.525 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:23:49.525 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-10T03:23:49.525 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T03:23:49.525 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T03:23:49.525 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T03:23:49.525 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T03:23:49.526 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:23:49.526 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:23:49.862 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T03:23:49.905 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T03:23:49.958 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T03:23:49.958 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T03:23:49.958 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T03:23:49.959 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T03:23:49.959 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T03:23:49.959 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-09-10T03:23:49.959 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi112.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T03:23:49.959 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T03:23:50.017 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T03:23:50.018 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T03:23:50.018 DEBUG:teuthology.orchestra.run.smithi112:> sudo rpm -qi grub2-tools 2024-09-10T03:23:50.058 INFO:teuthology.orchestra.run.smithi112.stdout:Name : grub2-tools 2024-09-10T03:23:50.058 INFO:teuthology.orchestra.run.smithi112.stdout:Epoch : 1 2024-09-10T03:23:50.058 INFO:teuthology.orchestra.run.smithi112.stdout:Version : 2.06 2024-09-10T03:23:50.059 INFO:teuthology.orchestra.run.smithi112.stdout:Release : 82.el9 2024-09-10T03:23:50.059 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: x86_64 2024-09-10T03:23:50.059 INFO:teuthology.orchestra.run.smithi112.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T03:23:50.059 INFO:teuthology.orchestra.run.smithi112.stdout:Group : Unspecified 2024-09-10T03:23:50.059 INFO:teuthology.orchestra.run.smithi112.stdout:Size : 8274670 2024-09-10T03:23:50.059 INFO:teuthology.orchestra.run.smithi112.stdout:License : GPLv3+ 2024-09-10T03:23:50.059 INFO:teuthology.orchestra.run.smithi112.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T03:23:50.059 INFO:teuthology.orchestra.run.smithi112.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T03:23:50.060 INFO:teuthology.orchestra.run.smithi112.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T03:23:50.060 INFO:teuthology.orchestra.run.smithi112.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T03:23:50.060 INFO:teuthology.orchestra.run.smithi112.stdout:Packager : builder@centos.org 2024-09-10T03:23:50.060 INFO:teuthology.orchestra.run.smithi112.stdout:Vendor : CentOS 2024-09-10T03:23:50.060 INFO:teuthology.orchestra.run.smithi112.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T03:23:50.060 INFO:teuthology.orchestra.run.smithi112.stdout:Summary : Support tools for GRUB. 2024-09-10T03:23:50.060 INFO:teuthology.orchestra.run.smithi112.stdout:Description : 2024-09-10T03:23:50.060 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:23:50.060 INFO:teuthology.orchestra.run.smithi112.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T03:23:50.060 INFO:teuthology.orchestra.run.smithi112.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T03:23:50.061 INFO:teuthology.orchestra.run.smithi112.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T03:23:50.061 INFO:teuthology.orchestra.run.smithi112.stdout:hardware devices. 2024-09-10T03:23:50.061 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:23:50.061 INFO:teuthology.orchestra.run.smithi112.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T03:23:50.062 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T03:23:50.062 INFO:teuthology.task.kernel:Updating grub on smithi112 to boot 5.14.0-505.el9.x86_64 2024-09-10T03:23:50.062 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T03:23:50.716 INFO:teuthology.orchestra.run.smithi112.stderr:Generating grub configuration file ... 2024-09-10T03:23:52.045 INFO:teuthology.orchestra.run.smithi112.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T03:23:52.066 INFO:teuthology.orchestra.run.smithi112.stderr:done 2024-09-10T03:23:52.069 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-09-10T03:23:52.084 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.4U3OIzbMUc 2024-09-10T03:23:52.085 DEBUG:teuthology.orchestra.run.smithi112:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4U3OIzbMUc 2024-09-10T03:23:52.157 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.4U3OIzbMUc 2024-09-10T03:23:52.307 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.4U3OIzbMUc is 6KB 2024-09-10T03:23:52.354 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.4U3OIzbMUc 2024-09-10T03:23:52.369 DEBUG:teuthology.orchestra.run.smithi112:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T03:23:52.431 INFO:teuthology.orchestra.run.smithi112.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T03:23:52.431 INFO:teuthology.orchestra.run.smithi112.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T03:23:52.431 INFO:teuthology.orchestra.run.smithi112.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T03:23:52.431 INFO:teuthology.orchestra.run.smithi112.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T03:23:52.433 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T03:23:52.610 DEBUG:teuthology.orchestra.run.smithi112:> sudo shutdown -r now 2024-09-10T03:24:22.613 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-09-10T03:24:22.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-10T03:24:22.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T03:24:28.092 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T03:24:28.131 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T03:24:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T03:24:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T03:24:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T03:24:28.559 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T03:24:29.419 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T03:24:29.419 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:24:29.420 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-10T03:24:29.420 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T03:24:29.420 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T03:24:29.420 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T03:24:29.420 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T03:24:29.420 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:24:29.420 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:24:29.757 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T03:24:29.851 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T03:24:29.851 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T03:24:29.851 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T03:24:29.851 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T03:24:29.852 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T03:24:29.852 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-09-10T03:24:29.852 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi169.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T03:24:29.852 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T03:24:29.911 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T03:24:29.912 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T03:24:29.912 DEBUG:teuthology.orchestra.run.smithi169:> sudo rpm -qi grub2-tools 2024-09-10T03:24:29.952 INFO:teuthology.orchestra.run.smithi169.stdout:Name : grub2-tools 2024-09-10T03:24:29.953 INFO:teuthology.orchestra.run.smithi169.stdout:Epoch : 1 2024-09-10T03:24:29.953 INFO:teuthology.orchestra.run.smithi169.stdout:Version : 2.06 2024-09-10T03:24:29.953 INFO:teuthology.orchestra.run.smithi169.stdout:Release : 82.el9 2024-09-10T03:24:29.953 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: x86_64 2024-09-10T03:24:29.953 INFO:teuthology.orchestra.run.smithi169.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T03:24:29.953 INFO:teuthology.orchestra.run.smithi169.stdout:Group : Unspecified 2024-09-10T03:24:29.953 INFO:teuthology.orchestra.run.smithi169.stdout:Size : 8274670 2024-09-10T03:24:29.953 INFO:teuthology.orchestra.run.smithi169.stdout:License : GPLv3+ 2024-09-10T03:24:29.953 INFO:teuthology.orchestra.run.smithi169.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T03:24:29.953 INFO:teuthology.orchestra.run.smithi169.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T03:24:29.953 INFO:teuthology.orchestra.run.smithi169.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T03:24:29.953 INFO:teuthology.orchestra.run.smithi169.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T03:24:29.953 INFO:teuthology.orchestra.run.smithi169.stdout:Packager : builder@centos.org 2024-09-10T03:24:29.954 INFO:teuthology.orchestra.run.smithi169.stdout:Vendor : CentOS 2024-09-10T03:24:29.954 INFO:teuthology.orchestra.run.smithi169.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T03:24:29.954 INFO:teuthology.orchestra.run.smithi169.stdout:Summary : Support tools for GRUB. 2024-09-10T03:24:29.954 INFO:teuthology.orchestra.run.smithi169.stdout:Description : 2024-09-10T03:24:29.954 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:24:29.954 INFO:teuthology.orchestra.run.smithi169.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T03:24:29.954 INFO:teuthology.orchestra.run.smithi169.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T03:24:29.954 INFO:teuthology.orchestra.run.smithi169.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T03:24:29.954 INFO:teuthology.orchestra.run.smithi169.stdout:hardware devices. 2024-09-10T03:24:29.954 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:24:29.954 INFO:teuthology.orchestra.run.smithi169.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T03:24:29.955 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T03:24:29.955 INFO:teuthology.task.kernel:Updating grub on smithi169 to boot 5.14.0-505.el9.x86_64 2024-09-10T03:24:29.955 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T03:24:30.605 INFO:teuthology.orchestra.run.smithi169.stderr:Generating grub configuration file ... 2024-09-10T03:24:31.907 INFO:teuthology.orchestra.run.smithi169.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T03:24:31.938 INFO:teuthology.orchestra.run.smithi169.stderr:done 2024-09-10T03:24:31.940 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-09-10T03:24:31.955 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.h70wEVzhTV 2024-09-10T03:24:31.955 DEBUG:teuthology.orchestra.run.smithi169:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.h70wEVzhTV 2024-09-10T03:24:32.027 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.h70wEVzhTV 2024-09-10T03:24:32.162 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.h70wEVzhTV is 6KB 2024-09-10T03:24:32.211 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.h70wEVzhTV 2024-09-10T03:24:32.226 DEBUG:teuthology.orchestra.run.smithi169:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T03:24:32.293 INFO:teuthology.orchestra.run.smithi169.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T03:24:32.293 INFO:teuthology.orchestra.run.smithi169.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T03:24:32.293 INFO:teuthology.orchestra.run.smithi169.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T03:24:32.293 INFO:teuthology.orchestra.run.smithi169.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T03:24:32.295 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T03:24:32.438 DEBUG:teuthology.orchestra.run.smithi169:> sudo shutdown -r now 2024-09-10T03:24:41.149 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-09-10T03:24:50.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-10T03:24:50.159 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T03:24:53.217 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-09-10T03:25:02.445 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-09-10T03:25:02.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T03:25:02.446 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T03:25:05.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-10T03:25:05.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T03:25:36.957 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T03:25:37.215 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-09-10T03:25:37.638 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-10T03:25:37.639 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-505.el9.x86_64"... 2024-09-10T03:25:37.639 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-09-10T03:25:37.653 INFO:teuthology.orchestra.run.smithi112.stdout:5.14.0-505.el9.x86_64 2024-09-10T03:25:37.653 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T03:25:37.653 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T03:25:37.653 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T03:25:38.654 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-09-10T03:25:38.654 DEBUG:teuthology.orchestra.run.smithi112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T03:25:38.782 INFO:teuthology.orchestra.run.smithi112.stdout:ttyS1 2024-09-10T03:25:38.804 DEBUG:teuthology.parallel:result is None 2024-09-10T03:25:45.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T03:25:45.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T03:26:04.349 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T03:26:16.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T03:26:16.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T03:26:18.730 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-09-10T03:26:19.124 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T03:26:19.124 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-505.el9.x86_64"... 2024-09-10T03:26:19.124 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-09-10T03:26:19.141 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-505.el9.x86_64 2024-09-10T03:26:19.141 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T03:26:19.141 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T03:26:19.141 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T03:26:20.142 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-09-10T03:26:20.142 DEBUG:teuthology.orchestra.run.smithi169:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T03:26:20.243 INFO:teuthology.orchestra.run.smithi169.stdout:ttyS1 2024-09-10T03:26:20.268 DEBUG:teuthology.parallel:result is None 2024-09-10T03:26:20.268 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-10T03:26:20.275 INFO:teuthology.task.internal:Creating test directory... 2024-09-10T03:26:20.276 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T03:26:20.279 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T03:26:20.295 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-10T03:26:20.301 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-10T03:26:20.311 INFO:teuthology.task.internal:Creating archive directory... 2024-09-10T03:26:20.311 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T03:26:20.338 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T03:26:20.396 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-10T03:26:20.403 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-10T03:26:20.403 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T03:26:20.406 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T03:26:20.439 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T03:26:20.444 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T03:26:20.451 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T03:26:20.454 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T03:26:20.455 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-10T03:26:20.462 INFO:teuthology.task.internal:Configuring sudo... 2024-09-10T03:26:20.462 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T03:26:20.496 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T03:26:20.522 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-10T03:26:20.531 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-10T03:26:20.531 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T03:26:20.564 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T03:26:20.579 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T03:26:20.663 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T03:26:20.710 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T03:26:20.793 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T03:26:20.835 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T03:26:20.835 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T03:26:20.901 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T03:26:20.944 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T03:26:20.992 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T03:26:21.074 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T03:26:21.114 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-10T03:26:21.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T03:26:21.180 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2024-09-10T03:26:21.183 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2024-09-10T03:26:21.232 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T03:26:21.262 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T03:26:21.616 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-10T03:26:21.624 INFO:teuthology.task.internal:Starting timer... 2024-09-10T03:26:21.624 INFO:teuthology.run_tasks:Running task pcp... 2024-09-10T03:26:21.652 INFO:teuthology.run_tasks:Running task selinux... 2024-09-10T03:26:21.667 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-10T03:26:21.668 DEBUG:teuthology.orchestra.run.smithi112:> sudo service auditd rotate 2024-09-10T03:26:21.736 INFO:teuthology.orchestra.run.smithi112.stdout:Rotating logs: 2024-09-10T03:26:21.738 DEBUG:teuthology.orchestra.run.smithi169:> sudo service auditd rotate 2024-09-10T03:26:21.801 INFO:teuthology.orchestra.run.smithi169.stdout:Rotating logs: 2024-09-10T03:26:21.803 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-10T03:26:21.803 DEBUG:teuthology.orchestra.run.smithi112:> /usr/sbin/getenforce 2024-09-10T03:26:21.836 INFO:teuthology.orchestra.run.smithi112.stdout:Permissive 2024-09-10T03:26:21.836 DEBUG:teuthology.orchestra.run.smithi169:> /usr/sbin/getenforce 2024-09-10T03:26:21.860 INFO:teuthology.orchestra.run.smithi169.stdout:Permissive 2024-09-10T03:26:21.860 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi112.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi169.front.sepia.ceph.com': 'permissive'} 2024-09-10T03:26:21.860 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-10T03:26:21.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T03:26:21.902 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-10T03:26:21.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T03:26:21.926 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-10T03:26:21.927 DEBUG:teuthology.orchestra.run.smithi112:> sudo /usr/sbin/setenforce permissive 2024-09-10T03:26:21.971 DEBUG:teuthology.orchestra.run.smithi169:> sudo /usr/sbin/setenforce permissive 2024-09-10T03:26:22.024 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-10T03:26:22.034 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-10T03:26:22.037 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-10T03:26:22.145 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-10T03:26:22.156 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-10T03:26:22.157 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi112.front.sepia.ceph.com,smithi169.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-10T03:30:03.458 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi112.front.sepia.ceph.com'), Remote(name='ubuntu@smithi169.front.sepia.ceph.com')] 2024-09-10T03:30:03.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-10T03:30:03.461 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T03:30:03.544 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-09-10T03:30:03.624 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-10T03:30:03.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T03:30:03.625 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T03:30:03.713 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-09-10T03:30:03.793 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T03:30:03.793 INFO:teuthology.run_tasks:Running task clock... 2024-09-10T03:30:03.806 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-10T03:30:03.806 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T03:30:03.807 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T03:30:03.810 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T03:30:03.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T03:30:03.837 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T03:30:03.852 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T03:30:03.864 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T03:30:03.877 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: ntpd: command not found 2024-09-10T03:30:03.879 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T03:30:03.889 INFO:teuthology.orchestra.run.smithi112.stdout:506 Cannot talk to daemon 2024-09-10T03:30:03.902 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T03:30:03.903 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: ntpd: command not found 2024-09-10T03:30:03.914 INFO:teuthology.orchestra.run.smithi169.stdout:506 Cannot talk to daemon 2024-09-10T03:30:03.916 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T03:30:03.928 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T03:30:03.942 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T03:30:03.975 INFO:teuthology.orchestra.run.smithi112.stderr:bash: line 1: ntpq: command not found 2024-09-10T03:30:03.978 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T03:30:03.978 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2024-09-10T03:30:03.978 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T03:30:03.978 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T03:30:03.978 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T03:30:03.978 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T03:30:03.996 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-09-10T03:30:04.000 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T03:30:04.000 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-09-10T03:30:04.000 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T03:30:04.000 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T03:30:04.000 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T03:30:04.000 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T03:30:04.001 INFO:teuthology.run_tasks:Running task install... 2024-09-10T03:30:04.009 DEBUG:teuthology.task.install:project ceph 2024-09-10T03:30:04.009 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3a5e3f16720f432aea965c40a8f6eb480095ec6c'}} 2024-09-10T03:30:04.009 DEBUG:teuthology.task.install:config {'branch': 'squid', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '3a5e3f16720f432aea965c40a8f6eb480095ec6c'} 2024-09-10T03:30:04.010 INFO:teuthology.task.install:Using flavor: default 2024-09-10T03:30:04.015 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-10T03:30:04.015 INFO:teuthology.task.install:extra packages: [] 2024-09-10T03:30:04.015 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3a5e3f16720f432aea965c40a8f6eb480095ec6c', 'tag': None, 'wait_for_package': False} 2024-09-10T03:30:04.016 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T03:30:04.016 INFO:teuthology.packaging:ref: None 2024-09-10T03:30:04.016 INFO:teuthology.packaging:tag: None 2024-09-10T03:30:04.016 INFO:teuthology.packaging:branch: squid 2024-09-10T03:30:04.016 INFO:teuthology.packaging:sha1: 3a5e3f16720f432aea965c40a8f6eb480095ec6c 2024-09-10T03:30:04.016 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-10T03:30:04.018 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3a5e3f16720f432aea965c40a8f6eb480095ec6c', 'tag': None, 'wait_for_package': False} 2024-09-10T03:30:04.018 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T03:30:04.019 INFO:teuthology.packaging:ref: None 2024-09-10T03:30:04.019 INFO:teuthology.packaging:tag: None 2024-09-10T03:30:04.019 INFO:teuthology.packaging:branch: squid 2024-09-10T03:30:04.019 INFO:teuthology.packaging:sha1: 3a5e3f16720f432aea965c40a8f6eb480095ec6c 2024-09-10T03:30:04.019 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-10T03:30:04.236 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/ 2024-09-10T03:30:04.237 INFO:teuthology.task.install.rpm:Package version is 19.1.1-298.gb53b6d3e 2024-09-10T03:30:04.247 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/ 2024-09-10T03:30:04.248 INFO:teuthology.task.install.rpm:Package version is 19.1.1-298.gb53b6d3e 2024-09-10T03:30:04.351 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T03:30:04.351 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-10T03:30:04.351 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T03:30:04.357 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T03:30:04.358 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T03:30:04.358 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T03:30:04.377 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-10T03:30:04.377 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T03:30:04.377 INFO:teuthology.packaging:ref: None 2024-09-10T03:30:04.377 INFO:teuthology.packaging:tag: None 2024-09-10T03:30:04.377 INFO:teuthology.packaging:branch: squid 2024-09-10T03:30:04.377 INFO:teuthology.packaging:sha1: 3a5e3f16720f432aea965c40a8f6eb480095ec6c 2024-09-10T03:30:04.378 DEBUG:teuthology.orchestra.run.smithi169:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T03:30:04.383 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-10T03:30:04.383 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T03:30:04.383 INFO:teuthology.packaging:ref: None 2024-09-10T03:30:04.383 INFO:teuthology.packaging:tag: None 2024-09-10T03:30:04.383 INFO:teuthology.packaging:branch: squid 2024-09-10T03:30:04.383 INFO:teuthology.packaging:sha1: 3a5e3f16720f432aea965c40a8f6eb480095ec6c 2024-09-10T03:30:04.384 DEBUG:teuthology.orchestra.run.smithi112:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T03:30:04.443 DEBUG:teuthology.orchestra.run.smithi169:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T03:30:04.449 DEBUG:teuthology.orchestra.run.smithi112:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T03:30:04.518 DEBUG:teuthology.orchestra.run.smithi169:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T03:30:04.524 DEBUG:teuthology.orchestra.run.smithi112:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T03:30:04.583 INFO:teuthology.orchestra.run.smithi169.stdout:check_obsoletes = 1 2024-09-10T03:30:04.584 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum clean all 2024-09-10T03:30:04.587 INFO:teuthology.orchestra.run.smithi112.stdout:check_obsoletes = 1 2024-09-10T03:30:04.588 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum clean all 2024-09-10T03:30:04.814 INFO:teuthology.orchestra.run.smithi112.stdout:45 files removed 2024-09-10T03:30:04.818 INFO:teuthology.orchestra.run.smithi169.stdout:45 files removed 2024-09-10T03:30:04.841 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-radosgw 2024-09-10T03:30:04.845 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-radosgw 2024-09-10T03:30:05.388 INFO:teuthology.orchestra.run.smithi112.stdout:ceph packages for x86_64 322 kB/s | 84 kB 00:00 2024-09-10T03:30:05.398 INFO:teuthology.orchestra.run.smithi169.stdout:ceph packages for x86_64 317 kB/s | 84 kB 00:00 2024-09-10T03:30:05.666 INFO:teuthology.orchestra.run.smithi169.stdout:ceph noarch packages 52 kB/s | 12 kB 00:00 2024-09-10T03:30:05.671 INFO:teuthology.orchestra.run.smithi112.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-10T03:30:05.931 INFO:teuthology.orchestra.run.smithi112.stdout:ceph source packages 8.5 kB/s | 1.9 kB 00:00 2024-09-10T03:30:05.932 INFO:teuthology.orchestra.run.smithi169.stdout:ceph source packages 8.2 kB/s | 1.9 kB 00:00 2024-09-10T03:30:07.483 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.2 MB 00:01 2024-09-10T03:30:07.697 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.2 MB 00:01 2024-09-10T03:30:13.051 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - AppStream 4.8 MB/s | 20 MB 00:04 2024-09-10T03:30:16.294 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-10T03:30:18.691 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.5 MB 00:01 2024-09-10T03:30:20.150 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 19 kB 00:00 2024-09-10T03:30:20.762 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 9.6 MB/s | 6.5 MB 00:00 2024-09-10T03:30:22.202 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 86 kB/s | 19 kB 00:00 2024-09-10T03:30:24.190 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 5.8 MB/s | 23 MB 00:03 2024-09-10T03:30:29.620 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-09-10T03:30:32.021 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:30:32.082 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:30:32.082 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-10T03:30:32.082 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:30:32.082 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-10T03:30:32.082 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 11 M 2024-09-10T03:30:32.083 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrading: 2024-09-10T03:30:32.083 INFO:teuthology.orchestra.run.smithi112.stdout: librados2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.4 M 2024-09-10T03:30:32.083 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.2 M 2024-09-10T03:30:32.083 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-10T03:30:32.083 INFO:teuthology.orchestra.run.smithi112.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-10T03:30:32.083 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 5.4 M 2024-09-10T03:30:32.083 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 21 M 2024-09-10T03:30:32.083 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 25 k 2024-09-10T03:30:32.083 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T03:30:32.083 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-10T03:30:32.083 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-10T03:30:32.083 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 752 k 2024-09-10T03:30:32.083 INFO:teuthology.orchestra.run.smithi112.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T03:30:32.084 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T03:30:32.084 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T03:30:32.084 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 506 k 2024-09-10T03:30:32.084 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T03:30:32.084 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 5.3 M 2024-09-10T03:30:32.084 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T03:30:32.084 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T03:30:32.084 INFO:teuthology.orchestra.run.smithi112.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-10T03:30:32.084 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 45 k 2024-09-10T03:30:32.084 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 136 k 2024-09-10T03:30:32.084 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 162 k 2024-09-10T03:30:32.084 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T03:30:32.085 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 321 k 2024-09-10T03:30:32.085 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 301 k 2024-09-10T03:30:32.085 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 100 k 2024-09-10T03:30:32.085 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-10T03:30:32.085 INFO:teuthology.orchestra.run.smithi112.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-10T03:30:32.085 INFO:teuthology.orchestra.run.smithi112.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-10T03:30:32.085 INFO:teuthology.orchestra.run.smithi112.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T03:30:32.085 INFO:teuthology.orchestra.run.smithi112.stdout:Installing weak dependencies: 2024-09-10T03:30:32.085 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-10T03:30:32.085 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:30:32.085 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-10T03:30:32.085 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:30:32.085 INFO:teuthology.orchestra.run.smithi112.stdout:Install 30 Packages 2024-09-10T03:30:32.085 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrade 2 Packages 2024-09-10T03:30:32.086 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:30:32.086 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 60 M 2024-09-10T03:30:32.086 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-10T03:30:33.871 INFO:teuthology.orchestra.run.smithi112.stdout:(1/32): ceph-common-19.1.1-298.gb53b6d3e.el9.x8 19 MB/s | 21 MB 00:01 2024-09-10T03:30:33.946 INFO:teuthology.orchestra.run.smithi112.stdout:(2/32): ceph-base-19.1.1-298.gb53b6d3e.el9.x86_ 4.5 MB/s | 5.4 MB 00:01 2024-09-10T03:30:33.972 INFO:teuthology.orchestra.run.smithi112.stdout:(3/32): ceph-selinux-19.1.1-298.gb53b6d3e.el9.x 251 kB/s | 25 kB 00:00 2024-09-10T03:30:34.038 INFO:teuthology.orchestra.run.smithi112.stdout:(4/32): libcephfs2-19.1.1-298.gb53b6d3e.el9.x86 8.0 MB/s | 752 kB 00:00 2024-09-10T03:30:34.064 INFO:teuthology.orchestra.run.smithi112.stdout:(5/32): libradosstriper1-19.1.1-298.gb53b6d3e.e 5.4 MB/s | 506 kB 00:00 2024-09-10T03:30:34.256 INFO:teuthology.orchestra.run.smithi112.stdout:(6/32): librgw2-19.1.1-298.gb53b6d3e.el9.x86_64 25 MB/s | 5.3 MB 00:00 2024-09-10T03:30:34.281 INFO:teuthology.orchestra.run.smithi112.stdout:(7/32): python3-ceph-argparse-19.1.1-298.gb53b6 208 kB/s | 45 kB 00:00 2024-09-10T03:30:34.340 INFO:teuthology.orchestra.run.smithi112.stdout:(8/32): python3-ceph-common-19.1.1-298.gb53b6d3 1.6 MB/s | 136 kB 00:00 2024-09-10T03:30:34.365 INFO:teuthology.orchestra.run.smithi112.stdout:(9/32): python3-cephfs-19.1.1-298.gb53b6d3e.el9 1.9 MB/s | 162 kB 00:00 2024-09-10T03:30:34.482 INFO:teuthology.orchestra.run.smithi112.stdout:(10/32): ceph-radosgw-19.1.1-298.gb53b6d3e.el9. 6.1 MB/s | 11 MB 00:01 2024-09-10T03:30:34.507 INFO:teuthology.orchestra.run.smithi112.stdout:(11/32): python3-rbd-19.1.1-298.gb53b6d3e.el9.x 2.1 MB/s | 301 kB 00:00 2024-09-10T03:30:34.532 INFO:teuthology.orchestra.run.smithi112.stdout:(12/32): python3-rados-19.1.1-298.gb53b6d3e.el9 1.6 MB/s | 321 kB 00:00 2024-09-10T03:30:34.558 INFO:teuthology.orchestra.run.smithi112.stdout:(13/32): python3-rgw-19.1.1-298.gb53b6d3e.el9.x 1.3 MB/s | 100 kB 00:00 2024-09-10T03:30:34.727 INFO:teuthology.orchestra.run.smithi112.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 268 kB/s | 45 kB 00:00 2024-09-10T03:30:34.752 INFO:teuthology.orchestra.run.smithi112.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 437 kB/s | 107 kB 00:00 2024-09-10T03:30:34.777 INFO:teuthology.orchestra.run.smithi112.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 655 kB/s | 160 kB 00:00 2024-09-10T03:30:34.828 INFO:teuthology.orchestra.run.smithi112.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 847 kB/s | 42 kB 00:00 2024-09-10T03:30:34.894 INFO:teuthology.orchestra.run.smithi112.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-09-10T03:30:34.919 INFO:teuthology.orchestra.run.smithi112.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 662 kB 00:00 2024-09-10T03:30:34.945 INFO:teuthology.orchestra.run.smithi112.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-09-10T03:30:34.970 INFO:teuthology.orchestra.run.smithi112.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 476 kB/s | 36 kB 00:00 2024-09-10T03:30:34.996 INFO:teuthology.orchestra.run.smithi112.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 627 kB/s | 47 kB 00:00 2024-09-10T03:30:35.062 INFO:teuthology.orchestra.run.smithi112.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 501 kB/s | 26 kB 00:00 2024-09-10T03:30:35.087 INFO:teuthology.orchestra.run.smithi112.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.1 MB/s | 309 kB 00:00 2024-09-10T03:30:35.113 INFO:teuthology.orchestra.run.smithi112.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 968 kB/s | 49 kB 00:00 2024-09-10T03:30:35.138 INFO:teuthology.orchestra.run.smithi112.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-10T03:30:35.163 INFO:teuthology.orchestra.run.smithi112.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 7.6 MB/s | 191 kB 00:00 2024-09-10T03:30:35.205 INFO:teuthology.orchestra.run.smithi112.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-10T03:30:35.306 INFO:teuthology.orchestra.run.smithi112.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-10T03:30:35.406 INFO:teuthology.orchestra.run.smithi112.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-09-10T03:30:35.473 INFO:teuthology.orchestra.run.smithi112.stdout:(31/32): librados2-19.1.1-298.gb53b6d3e.el9.x86 13 MB/s | 3.4 MB 00:00 2024-09-10T03:30:35.564 INFO:teuthology.orchestra.run.smithi112.stdout:(32/32): librbd1-19.1.1-298.gb53b6d3e.el9.x86_6 12 MB/s | 3.2 MB 00:00 2024-09-10T03:30:35.569 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:30:35.569 INFO:teuthology.orchestra.run.smithi112.stdout:Total 17 MB/s | 60 MB 00:03 2024-09-10T03:30:35.669 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-10T03:30:35.705 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-10T03:30:35.705 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-10T03:30:36.379 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-10T03:30:36.380 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-10T03:30:37.874 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-10T03:30:37.948 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-10T03:30:37.978 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-10T03:30:38.003 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-10T03:30:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-10T03:30:38.051 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-10T03:30:38.253 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-10T03:30:38.306 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-10T03:30:38.681 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-10T03:30:38.746 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/34 2024-09-10T03:30:38.839 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/34 2024-09-10T03:30:38.928 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 8/34 2024-09-10T03:30:38.956 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/34 2024-09-10T03:30:39.010 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/34 2024-09-10T03:30:39.039 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-10T03:30:39.083 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x 11/34 2024-09-10T03:30:39.153 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 12/34 2024-09-10T03:30:39.174 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/34 2024-09-10T03:30:39.239 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/34 2024-09-10T03:30:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-10T03:30:39.383 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-10T03:30:39.416 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-10T03:30:39.457 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-10T03:30:39.497 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-10T03:30:39.526 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-10T03:30:39.550 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-10T03:30:39.875 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-10T03:30:39.922 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 22/34 2024-09-10T03:30:40.012 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 22/34 2024-09-10T03:30:40.050 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 23/34 2024-09-10T03:30:40.096 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-10T03:30:40.504 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 1.2 MB/s | 23 MB 00:18 2024-09-10T03:30:40.574 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-10T03:30:40.599 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 26/34 2024-09-10T03:30:40.670 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 26/34 2024-09-10T03:30:40.746 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 27/34 2024-09-10T03:30:40.776 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86 28/34 2024-09-10T03:30:43.013 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/34 2024-09-10T03:30:43.047 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/34 2024-09-10T03:30:43.638 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/34 2024-09-10T03:30:43.688 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 30/34 2024-09-10T03:30:43.776 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 30/34 2024-09-10T03:30:43.776 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T03:30:43.777 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T03:30:43.777 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:30:43.810 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 31/34 2024-09-10T03:30:45.968 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-10T03:30:48.345 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:30:48.356 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:30:48.356 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-10T03:30:48.356 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:30:48.356 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-10T03:30:48.356 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 11 M 2024-09-10T03:30:48.356 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrading: 2024-09-10T03:30:48.357 INFO:teuthology.orchestra.run.smithi169.stdout: librados2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.4 M 2024-09-10T03:30:48.357 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.2 M 2024-09-10T03:30:48.357 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-10T03:30:48.357 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-10T03:30:48.357 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 5.4 M 2024-09-10T03:30:48.357 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 21 M 2024-09-10T03:30:48.357 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 25 k 2024-09-10T03:30:48.357 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T03:30:48.357 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-10T03:30:48.357 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-10T03:30:48.357 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 752 k 2024-09-10T03:30:48.357 INFO:teuthology.orchestra.run.smithi169.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T03:30:48.358 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T03:30:48.358 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T03:30:48.358 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 506 k 2024-09-10T03:30:48.358 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T03:30:48.358 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 5.3 M 2024-09-10T03:30:48.358 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T03:30:48.358 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T03:30:48.358 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-10T03:30:48.358 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 45 k 2024-09-10T03:30:48.358 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 136 k 2024-09-10T03:30:48.358 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 162 k 2024-09-10T03:30:48.359 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T03:30:48.359 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 321 k 2024-09-10T03:30:48.359 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 301 k 2024-09-10T03:30:48.359 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 100 k 2024-09-10T03:30:48.359 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-10T03:30:48.359 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-10T03:30:48.359 INFO:teuthology.orchestra.run.smithi169.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-10T03:30:48.359 INFO:teuthology.orchestra.run.smithi169.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T03:30:48.359 INFO:teuthology.orchestra.run.smithi169.stdout:Installing weak dependencies: 2024-09-10T03:30:48.359 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-10T03:30:48.359 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:30:48.359 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-10T03:30:48.360 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:30:48.360 INFO:teuthology.orchestra.run.smithi169.stdout:Install 30 Packages 2024-09-10T03:30:48.360 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrade 2 Packages 2024-09-10T03:30:48.360 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:30:48.360 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 60 M 2024-09-10T03:30:48.360 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-10T03:30:49.120 INFO:teuthology.orchestra.run.smithi169.stdout:(1/32): ceph-base-19.1.1-298.gb53b6d3e.el9.x86_ 9.4 MB/s | 5.4 MB 00:00 2024-09-10T03:30:49.170 INFO:teuthology.orchestra.run.smithi169.stdout:(2/32): ceph-selinux-19.1.1-298.gb53b6d3e.el9.x 504 kB/s | 25 kB 00:00 2024-09-10T03:30:49.270 INFO:teuthology.orchestra.run.smithi169.stdout:(3/32): libcephfs2-19.1.1-298.gb53b6d3e.el9.x86 7.4 MB/s | 752 kB 00:00 2024-09-10T03:30:49.387 INFO:teuthology.orchestra.run.smithi169.stdout:(4/32): ceph-radosgw-19.1.1-298.gb53b6d3e.el9.x 13 MB/s | 11 MB 00:00 2024-09-10T03:30:49.421 INFO:teuthology.orchestra.run.smithi169.stdout:(5/32): libradosstriper1-19.1.1-298.gb53b6d3e.e 3.3 MB/s | 506 kB 00:00 2024-09-10T03:30:49.471 INFO:teuthology.orchestra.run.smithi169.stdout:(6/32): python3-ceph-argparse-19.1.1-298.gb53b6 899 kB/s | 45 kB 00:00 2024-09-10T03:30:49.563 INFO:teuthology.orchestra.run.smithi169.stdout:(7/32): python3-ceph-common-19.1.1-298.gb53b6d3 1.4 MB/s | 136 kB 00:00 2024-09-10T03:30:49.772 INFO:teuthology.orchestra.run.smithi169.stdout:(8/32): ceph-common-19.1.1-298.gb53b6d3e.el9.x8 17 MB/s | 21 MB 00:01 2024-09-10T03:30:49.797 INFO:teuthology.orchestra.run.smithi169.stdout:(9/32): python3-cephfs-19.1.1-298.gb53b6d3e.el9 694 kB/s | 162 kB 00:00 2024-09-10T03:30:49.831 INFO:teuthology.orchestra.run.smithi169.stdout:(10/32): python3-rados-19.1.1-298.gb53b6d3e.el9 5.4 MB/s | 321 kB 00:00 2024-09-10T03:30:49.864 INFO:teuthology.orchestra.run.smithi169.stdout:(11/32): python3-rbd-19.1.1-298.gb53b6d3e.el9.x 4.4 MB/s | 301 kB 00:00 2024-09-10T03:30:49.898 INFO:teuthology.orchestra.run.smithi169.stdout:(12/32): python3-rgw-19.1.1-298.gb53b6d3e.el9.x 1.5 MB/s | 100 kB 00:00 2024-09-10T03:30:49.982 INFO:teuthology.orchestra.run.smithi169.stdout:(13/32): librgw2-19.1.1-298.gb53b6d3e.el9.x86_6 9.0 MB/s | 5.3 MB 00:00 2024-09-10T03:30:50.100 INFO:teuthology.orchestra.run.smithi169.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 388 kB/s | 45 kB 00:00 2024-09-10T03:30:50.140 INFO:teuthology.orchestra.run.smithi169.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 387 kB/s | 107 kB 00:00 2024-09-10T03:30:50.174 INFO:teuthology.orchestra.run.smithi169.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 581 kB/s | 160 kB 00:00 2024-09-10T03:30:50.233 INFO:teuthology.orchestra.run.smithi169.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-10T03:30:50.258 INFO:teuthology.orchestra.run.smithi169.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 662 kB 00:00 2024-09-10T03:30:50.283 INFO:teuthology.orchestra.run.smithi169.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 388 kB/s | 42 kB 00:00 2024-09-10T03:30:50.350 INFO:teuthology.orchestra.run.smithi169.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-09-10T03:30:50.375 INFO:teuthology.orchestra.run.smithi169.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 306 kB/s | 36 kB 00:00 2024-09-10T03:30:50.401 INFO:teuthology.orchestra.run.smithi169.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 405 kB/s | 47 kB 00:00 2024-09-10T03:30:50.584 INFO:teuthology.orchestra.run.smithi169.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 155 kB/s | 26 kB 00:00 2024-09-10T03:30:50.610 INFO:teuthology.orchestra.run.smithi169.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-09-10T03:30:50.635 INFO:teuthology.orchestra.run.smithi169.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 964 kB/s | 49 kB 00:00 2024-09-10T03:30:50.660 INFO:teuthology.orchestra.run.smithi169.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-10T03:30:50.694 INFO:teuthology.orchestra.run.smithi169.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 14 MB/s | 839 kB 00:00 2024-09-10T03:30:50.719 INFO:teuthology.orchestra.run.smithi169.stdout:(28/32): re2-20211101-20.el9.x86_64.rpm 3.2 MB/s | 191 kB 00:00 2024-09-10T03:30:50.819 INFO:teuthology.orchestra.run.smithi169.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-10T03:30:50.928 INFO:teuthology.orchestra.run.smithi169.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 8.0 MB/s | 4.4 MB 00:00 2024-09-10T03:30:50.995 INFO:teuthology.orchestra.run.smithi169.stdout:(31/32): librados2-19.1.1-298.gb53b6d3e.el9.x86 12 MB/s | 3.4 MB 00:00 2024-09-10T03:30:51.104 INFO:teuthology.orchestra.run.smithi169.stdout:(32/32): librbd1-19.1.1-298.gb53b6d3e.el9.x86_6 11 MB/s | 3.2 MB 00:00 2024-09-10T03:30:51.109 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:30:51.109 INFO:teuthology.orchestra.run.smithi169.stdout:Total 22 MB/s | 60 MB 00:02 2024-09-10T03:30:51.214 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-10T03:30:51.251 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-10T03:30:51.251 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-10T03:30:51.925 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-10T03:30:51.925 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-10T03:30:53.394 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-10T03:30:53.477 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-10T03:30:53.509 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-10T03:30:53.534 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-10T03:30:53.550 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 31/34 2024-09-10T03:30:53.550 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /sys 2024-09-10T03:30:53.550 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /proc 2024-09-10T03:30:53.550 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /mnt 2024-09-10T03:30:53.550 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /var/tmp 2024-09-10T03:30:53.551 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /home 2024-09-10T03:30:53.551 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /root 2024-09-10T03:30:53.551 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /tmp 2024-09-10T03:30:53.551 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:30:53.556 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-10T03:30:53.578 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-10T03:30:53.760 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-10T03:30:53.812 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-10T03:30:54.143 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-10T03:30:54.252 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/34 2024-09-10T03:30:54.343 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/34 2024-09-10T03:30:54.439 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 8/34 2024-09-10T03:30:54.470 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/34 2024-09-10T03:30:54.532 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 32/34 2024-09-10T03:30:54.533 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/34 2024-09-10T03:30:54.558 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 32/34 2024-09-10T03:30:54.558 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T03:30:54.558 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T03:30:54.558 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T03:30:54.558 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T03:30:54.558 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:30:54.560 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-10T03:30:54.570 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-10T03:30:54.614 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x 11/34 2024-09-10T03:30:54.627 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-10T03:30:54.627 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-10T03:30:54.682 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 12/34 2024-09-10T03:30:54.706 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/34 2024-09-10T03:30:54.770 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/34 2024-09-10T03:30:54.804 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-10T03:30:54.912 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-10T03:30:54.939 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-10T03:30:54.988 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-10T03:30:55.029 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-10T03:30:55.057 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-10T03:30:55.090 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-10T03:30:55.402 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-10T03:30:55.445 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 22/34 2024-09-10T03:30:55.536 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 22/34 2024-09-10T03:30:55.569 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 23/34 2024-09-10T03:30:55.610 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-10T03:30:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-10T03:30:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/34 2024-09-10T03:30:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/34 2024-09-10T03:30:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/34 2024-09-10T03:30:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/34 2024-09-10T03:30:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 5/34 2024-09-10T03:30:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 6/34 2024-09-10T03:30:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/34 2024-09-10T03:30:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x 8/34 2024-09-10T03:30:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86 9/34 2024-09-10T03:30:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 10/34 2024-09-10T03:30:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 11/34 2024-09-10T03:30:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 12/34 2024-09-10T03:30:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/34 2024-09-10T03:30:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-10T03:30:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-10T03:30:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-10T03:30:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-10T03:30:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-10T03:30:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-10T03:30:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-10T03:30:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-10T03:30:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-10T03:30:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-10T03:30:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-10T03:30:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-10T03:30:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-10T03:30:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-10T03:30:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-10T03:30:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-10T03:30:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-10T03:30:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 31/34 2024-09-10T03:30:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-10T03:30:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 33/34 2024-09-10T03:30:56.089 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-10T03:30:56.114 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 26/34 2024-09-10T03:30:56.185 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 26/34 2024-09-10T03:30:56.254 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 27/34 2024-09-10T03:30:56.291 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86 28/34 2024-09-10T03:30:56.993 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-10T03:30:56.993 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:30:56.993 INFO:teuthology.orchestra.run.smithi112.stdout:Upgraded: 2024-09-10T03:30:56.993 INFO:teuthology.orchestra.run.smithi112.stdout: librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:30:56.993 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:30:56.993 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-10T03:30:56.994 INFO:teuthology.orchestra.run.smithi112.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-10T03:30:56.994 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:30:56.994 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:30:56.994 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:30:56.994 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:30:56.994 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T03:30:56.994 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-10T03:30:56.994 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-10T03:30:56.994 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:30:56.994 INFO:teuthology.orchestra.run.smithi112.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T03:30:56.994 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T03:30:56.994 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T03:30:56.994 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:30:56.995 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T03:30:56.995 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:30:56.995 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T03:30:56.995 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T03:30:56.995 INFO:teuthology.orchestra.run.smithi112.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-10T03:30:56.995 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:30:56.995 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:30:56.995 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:30:56.995 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T03:30:56.995 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:30:56.995 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:30:56.995 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:30:56.995 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-10T03:30:56.996 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-10T03:30:56.996 INFO:teuthology.orchestra.run.smithi112.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-10T03:30:56.996 INFO:teuthology.orchestra.run.smithi112.stdout: re2-1:20211101-20.el9.x86_64 2024-09-10T03:30:56.996 INFO:teuthology.orchestra.run.smithi112.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T03:30:56.996 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:30:56.996 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:30:57.330 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-test 2024-09-10T03:30:57.892 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:28 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:30:58.023 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:30:58.024 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:30:58.024 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-10T03:30:58.024 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:30:58.024 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-10T03:30:58.025 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 49 M 2024-09-10T03:30:58.025 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-10T03:30:58.025 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 163 k 2024-09-10T03:30:58.025 INFO:teuthology.orchestra.run.smithi112.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T03:30:58.025 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T03:30:58.025 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:30:58.025 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-10T03:30:58.025 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:30:58.025 INFO:teuthology.orchestra.run.smithi112.stdout:Install 4 Packages 2024-09-10T03:30:58.025 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:30:58.026 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 49 M 2024-09-10T03:30:58.026 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 209 M 2024-09-10T03:30:58.026 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-10T03:30:58.265 INFO:teuthology.orchestra.run.smithi112.stdout:(1/4): libcephsqlite-19.1.1-298.gb53b6d3e.el9.x 755 kB/s | 163 kB 00:00 2024-09-10T03:30:58.557 INFO:teuthology.orchestra.run.smithi112.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 598 kB/s | 304 kB 00:00 2024-09-10T03:30:58.607 INFO:teuthology.orchestra.run.smithi112.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 186 kB/s | 64 kB 00:00 2024-09-10T03:30:58.615 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/34 2024-09-10T03:30:58.696 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/34 2024-09-10T03:30:59.273 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/34 2024-09-10T03:30:59.316 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 30/34 2024-09-10T03:30:59.414 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 30/34 2024-09-10T03:30:59.414 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T03:30:59.414 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T03:30:59.414 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:30:59.442 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 31/34 2024-09-10T03:31:00.432 INFO:teuthology.orchestra.run.smithi112.stdout:(4/4): ceph-test-19.1.1-298.gb53b6d3e.el9.x86_6 20 MB/s | 49 MB 00:02 2024-09-10T03:31:00.434 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:31:00.435 INFO:teuthology.orchestra.run.smithi112.stdout:Total 20 MB/s | 49 MB 00:02 2024-09-10T03:31:00.463 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-10T03:31:00.483 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-10T03:31:00.483 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-10T03:31:00.864 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-10T03:31:00.865 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-10T03:31:01.393 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-10T03:31:01.476 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-10T03:31:01.523 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-10T03:31:01.550 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/4 2024-09-10T03:31:06.296 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/4 2024-09-10T03:31:06.390 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/4 2024-09-10T03:31:08.647 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/4 2024-09-10T03:31:08.648 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/4 2024-09-10T03:31:08.648 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/4 2024-09-10T03:31:08.648 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-10T03:31:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-10T03:31:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:08.995 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-10T03:31:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T03:31:08.995 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T03:31:08.996 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:08.996 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:31:09.249 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph 2024-09-10T03:31:09.549 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 31/34 2024-09-10T03:31:09.717 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /sys 2024-09-10T03:31:09.718 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /proc 2024-09-10T03:31:09.718 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /mnt 2024-09-10T03:31:09.718 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /var/tmp 2024-09-10T03:31:09.718 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /home 2024-09-10T03:31:09.718 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /root 2024-09-10T03:31:09.718 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /tmp 2024-09-10T03:31:09.718 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:09.827 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:40 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:31:09.975 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:31:09.978 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:09.978 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-10T03:31:09.978 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:09.978 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-10T03:31:09.978 INFO:teuthology.orchestra.run.smithi112.stdout: ceph x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 6.5 k 2024-09-10T03:31:09.978 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-10T03:31:09.978 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 2.4 M 2024-09-10T03:31:09.978 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 1.5 M 2024-09-10T03:31:09.978 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 249 k 2024-09-10T03:31:09.978 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 4.7 M 2024-09-10T03:31:09.979 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 17 M 2024-09-10T03:31:09.979 INFO:teuthology.orchestra.run.smithi112.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-10T03:31:09.979 INFO:teuthology.orchestra.run.smithi112.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-10T03:31:09.979 INFO:teuthology.orchestra.run.smithi112.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-10T03:31:09.979 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T03:31:09.979 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T03:31:09.979 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T03:31:09.979 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T03:31:09.979 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T03:31:09.979 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T03:31:09.979 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T03:31:09.979 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-10T03:31:09.980 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T03:31:09.980 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T03:31:09.980 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T03:31:09.980 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T03:31:09.980 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T03:31:09.980 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T03:31:09.980 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T03:31:09.980 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T03:31:09.980 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T03:31:09.980 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T03:31:09.980 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T03:31:09.981 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T03:31:09.981 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T03:31:09.981 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T03:31:09.981 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T03:31:09.981 INFO:teuthology.orchestra.run.smithi112.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T03:31:09.981 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T03:31:09.981 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T03:31:09.981 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T03:31:09.981 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T03:31:09.981 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:09.982 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-10T03:31:09.982 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:09.982 INFO:teuthology.orchestra.run.smithi112.stdout:Install 37 Packages 2024-09-10T03:31:09.982 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:09.982 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 30 M 2024-09-10T03:31:09.982 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 108 M 2024-09-10T03:31:09.982 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-10T03:31:10.482 INFO:teuthology.orchestra.run.smithi112.stdout:(1/37): ceph-19.1.1-298.gb53b6d3e.el9.x86_64.rp 43 kB/s | 6.5 kB 00:00 2024-09-10T03:31:10.535 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 32/34 2024-09-10T03:31:10.560 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 32/34 2024-09-10T03:31:10.561 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T03:31:10.561 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T03:31:10.561 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T03:31:10.561 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T03:31:10.561 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:10.562 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-10T03:31:10.638 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-10T03:31:10.638 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-10T03:31:10.674 INFO:teuthology.orchestra.run.smithi112.stdout:(2/37): ceph-mgr-19.1.1-298.gb53b6d3e.el9.x86_6 4.4 MB/s | 1.5 MB 00:00 2024-09-10T03:31:10.724 INFO:teuthology.orchestra.run.smithi112.stdout:(3/37): ceph-mds-19.1.1-298.gb53b6d3e.el9.x86_6 6.1 MB/s | 2.4 MB 00:00 2024-09-10T03:31:10.899 INFO:teuthology.orchestra.run.smithi112.stdout:(4/37): ceph-mon-19.1.1-298.gb53b6d3e.el9.x86_6 11 MB/s | 4.7 MB 00:00 2024-09-10T03:31:10.924 INFO:teuthology.orchestra.run.smithi112.stdout:(5/37): ceph-mgr-modules-core-19.1.1-298.gb53b6 1.2 MB/s | 249 kB 00:00 2024-09-10T03:31:11.100 INFO:teuthology.orchestra.run.smithi112.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-10T03:31:11.158 INFO:teuthology.orchestra.run.smithi112.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-10T03:31:11.217 INFO:teuthology.orchestra.run.smithi112.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-09-10T03:31:11.251 INFO:teuthology.orchestra.run.smithi112.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 3.8 MB/s | 1.2 MB 00:00 2024-09-10T03:31:11.276 INFO:teuthology.orchestra.run.smithi112.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-09-10T03:31:11.310 INFO:teuthology.orchestra.run.smithi112.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-10T03:31:11.468 INFO:teuthology.orchestra.run.smithi112.stdout:(12/37): ceph-osd-19.1.1-298.gb53b6d3e.el9.x86_ 21 MB/s | 17 MB 00:00 2024-09-10T03:31:11.785 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-10T03:31:11.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/34 2024-09-10T03:31:11.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/34 2024-09-10T03:31:11.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/34 2024-09-10T03:31:11.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/34 2024-09-10T03:31:11.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 5/34 2024-09-10T03:31:11.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 6/34 2024-09-10T03:31:11.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/34 2024-09-10T03:31:11.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x 8/34 2024-09-10T03:31:11.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86 9/34 2024-09-10T03:31:11.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 10/34 2024-09-10T03:31:11.786 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 11/34 2024-09-10T03:31:11.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 12/34 2024-09-10T03:31:11.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/34 2024-09-10T03:31:11.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-10T03:31:11.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-10T03:31:11.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-10T03:31:11.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-10T03:31:11.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-10T03:31:11.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-10T03:31:11.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-10T03:31:11.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-10T03:31:11.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-10T03:31:11.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-10T03:31:11.787 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-10T03:31:11.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-10T03:31:11.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-10T03:31:11.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-10T03:31:11.789 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-10T03:31:11.789 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-10T03:31:11.789 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-10T03:31:11.789 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 31/34 2024-09-10T03:31:11.789 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-10T03:31:11.789 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 33/34 2024-09-10T03:31:12.052 INFO:teuthology.orchestra.run.smithi112.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 60 kB/s | 35 kB 00:00 2024-09-10T03:31:12.160 INFO:teuthology.orchestra.run.smithi112.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 202 kB/s | 172 kB 00:00 2024-09-10T03:31:12.186 INFO:teuthology.orchestra.run.smithi112.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 207 kB/s | 188 kB 00:00 2024-09-10T03:31:12.736 INFO:teuthology.orchestra.run.smithi112.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 113 kB/s | 77 kB 00:00 2024-09-10T03:31:12.761 INFO:teuthology.orchestra.run.smithi112.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 72 kB/s | 42 kB 00:00 2024-09-10T03:31:12.803 INFO:teuthology.orchestra.run.smithi112.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-09-10T03:31:12.829 INFO:teuthology.orchestra.run.smithi112.stdout:(19/37): python3-pytz-2021.1-5.el9.noarch.rpm 76 kB/s | 51 kB 00:00 2024-09-10T03:31:12.848 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-10T03:31:12.849 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:12.849 INFO:teuthology.orchestra.run.smithi169.stdout:Upgraded: 2024-09-10T03:31:12.849 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:12.849 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:12.849 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-10T03:31:12.849 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-10T03:31:12.849 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:12.849 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:12.849 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:12.849 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:12.849 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T03:31:12.849 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-10T03:31:12.850 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-10T03:31:12.850 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:12.850 INFO:teuthology.orchestra.run.smithi169.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T03:31:12.850 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T03:31:12.850 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T03:31:12.850 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:12.850 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T03:31:12.850 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:12.850 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T03:31:12.850 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T03:31:12.850 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-10T03:31:12.850 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:12.850 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:12.850 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:12.851 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T03:31:12.851 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:12.851 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:12.851 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:12.851 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-10T03:31:12.851 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-10T03:31:12.851 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-10T03:31:12.851 INFO:teuthology.orchestra.run.smithi169.stdout: re2-1:20211101-20.el9.x86_64 2024-09-10T03:31:12.851 INFO:teuthology.orchestra.run.smithi169.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T03:31:12.851 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:12.851 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:31:12.853 INFO:teuthology.orchestra.run.smithi112.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-09-10T03:31:12.879 INFO:teuthology.orchestra.run.smithi112.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 584 kB/s | 43 kB 00:00 2024-09-10T03:31:12.921 INFO:teuthology.orchestra.run.smithi112.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 401 kB/s | 11 kB 00:00 2024-09-10T03:31:12.946 INFO:teuthology.orchestra.run.smithi112.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 766 kB/s | 18 kB 00:00 2024-09-10T03:31:12.971 INFO:teuthology.orchestra.run.smithi112.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 929 kB/s | 23 kB 00:00 2024-09-10T03:31:12.996 INFO:teuthology.orchestra.run.smithi112.stdout:(25/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-10T03:31:13.022 INFO:teuthology.orchestra.run.smithi112.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-10T03:31:13.047 INFO:teuthology.orchestra.run.smithi112.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-10T03:31:13.072 INFO:teuthology.orchestra.run.smithi112.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 926 kB/s | 46 kB 00:00 2024-09-10T03:31:13.089 INFO:teuthology.orchestra.run.smithi112.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-10T03:31:13.114 INFO:teuthology.orchestra.run.smithi112.stdout:(30/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-10T03:31:13.140 INFO:teuthology.orchestra.run.smithi112.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-10T03:31:13.165 INFO:teuthology.orchestra.run.smithi112.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-10T03:31:13.190 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-test 2024-09-10T03:31:13.191 INFO:teuthology.orchestra.run.smithi112.stdout:(33/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-09-10T03:31:13.215 INFO:teuthology.orchestra.run.smithi112.stdout:(34/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-10T03:31:13.241 INFO:teuthology.orchestra.run.smithi112.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 808 kB/s | 20 kB 00:00 2024-09-10T03:31:13.266 INFO:teuthology.orchestra.run.smithi112.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-10T03:31:13.291 INFO:teuthology.orchestra.run.smithi112.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-09-10T03:31:13.296 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:31:13.296 INFO:teuthology.orchestra.run.smithi112.stdout:Total 9.1 MB/s | 30 MB 00:03 2024-09-10T03:31:13.451 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-10T03:31:13.491 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-10T03:31:13.491 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-10T03:31:13.763 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:28 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:31:13.896 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:31:13.897 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:31:13.897 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-10T03:31:13.897 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:31:13.897 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-10T03:31:13.897 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 49 M 2024-09-10T03:31:13.897 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-10T03:31:13.898 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 163 k 2024-09-10T03:31:13.898 INFO:teuthology.orchestra.run.smithi169.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T03:31:13.898 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T03:31:13.898 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:13.898 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-10T03:31:13.898 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:31:13.898 INFO:teuthology.orchestra.run.smithi169.stdout:Install 4 Packages 2024-09-10T03:31:13.898 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:13.899 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 49 M 2024-09-10T03:31:13.899 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 209 M 2024-09-10T03:31:13.899 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-10T03:31:13.952 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-10T03:31:13.952 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-10T03:31:14.159 INFO:teuthology.orchestra.run.smithi169.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.9 MB/s | 304 kB 00:00 2024-09-10T03:31:14.201 INFO:teuthology.orchestra.run.smithi169.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-10T03:31:14.226 INFO:teuthology.orchestra.run.smithi169.stdout:(3/4): libcephsqlite-19.1.1-298.gb53b6d3e.el9.x 719 kB/s | 163 kB 00:00 2024-09-10T03:31:15.297 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-10T03:31:15.378 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-10T03:31:15.411 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-10T03:31:15.438 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-10T03:31:15.508 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-10T03:31:15.543 INFO:teuthology.orchestra.run.smithi169.stdout:(4/4): ceph-test-19.1.1-298.gb53b6d3e.el9.x86_6 32 MB/s | 49 MB 00:01 2024-09-10T03:31:15.545 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-10T03:31:15.546 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:31:15.546 INFO:teuthology.orchestra.run.smithi169.stdout:Total 30 MB/s | 49 MB 00:01 2024-09-10T03:31:15.574 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-10T03:31:15.578 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-10T03:31:15.595 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-10T03:31:15.595 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-10T03:31:15.673 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-10T03:31:15.747 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-10T03:31:15.789 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-10T03:31:15.824 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-10T03:31:15.857 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-10T03:31:15.891 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-10T03:31:15.925 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-10T03:31:15.965 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-10T03:31:15.978 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-10T03:31:15.978 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-10T03:31:16.008 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-10T03:31:16.051 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-10T03:31:16.100 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-10T03:31:16.132 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-10T03:31:16.205 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-10T03:31:16.359 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-10T03:31:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-10T03:31:16.493 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-10T03:31:16.539 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-10T03:31:16.543 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-10T03:31:16.588 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-10T03:31:16.629 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-10T03:31:16.644 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-10T03:31:16.676 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-10T03:31:16.703 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/4 2024-09-10T03:31:16.828 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-10T03:31:16.877 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-10T03:31:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-10T03:31:17.067 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-10T03:31:17.109 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-10T03:31:17.225 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-10T03:31:17.382 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.n 32/37 2024-09-10T03:31:17.411 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 33/37 2024-09-10T03:31:17.438 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 33/37 2024-09-10T03:31:17.438 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T03:31:17.438 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T03:31:17.438 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T03:31:17.439 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T03:31:17.439 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:18.886 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 34/37 2024-09-10T03:31:18.915 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 34/37 2024-09-10T03:31:18.915 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T03:31:18.915 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T03:31:18.915 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T03:31:18.915 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T03:31:18.915 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:19.354 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 35/37 2024-09-10T03:31:19.378 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 35/37 2024-09-10T03:31:19.378 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T03:31:19.378 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T03:31:19.378 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T03:31:19.379 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T03:31:19.379 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:19.612 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 36/37 2024-09-10T03:31:19.636 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 36/37 2024-09-10T03:31:19.637 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T03:31:19.637 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T03:31:19.637 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T03:31:19.637 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T03:31:19.637 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:19.700 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 37/37 2024-09-10T03:31:21.329 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/4 2024-09-10T03:31:21.411 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/4 2024-09-10T03:31:21.496 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 37/37 2024-09-10T03:31:21.496 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/37 2024-09-10T03:31:21.496 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/37 2024-09-10T03:31:21.496 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/37 2024-09-10T03:31:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/37 2024-09-10T03:31:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 5/37 2024-09-10T03:31:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.n 6/37 2024-09-10T03:31:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-10T03:31:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-10T03:31:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-10T03:31:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-10T03:31:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-10T03:31:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-10T03:31:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-10T03:31:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-10T03:31:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-10T03:31:21.497 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-10T03:31:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-10T03:31:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-10T03:31:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-10T03:31:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-10T03:31:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-10T03:31:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-10T03:31:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-10T03:31:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-10T03:31:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-10T03:31:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-10T03:31:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-10T03:31:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-10T03:31:21.498 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-10T03:31:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-10T03:31:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-10T03:31:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-10T03:31:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-10T03:31:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-10T03:31:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-10T03:31:21.499 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-10T03:31:22.480 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-10T03:31:22.480 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:22.480 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-10T03:31:22.481 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:22.481 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:22.481 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:22.481 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-10T03:31:22.481 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:22.481 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:22.481 INFO:teuthology.orchestra.run.smithi112.stdout: lua-5.4.4-4.el9.x86_64 2024-09-10T03:31:22.481 INFO:teuthology.orchestra.run.smithi112.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-10T03:31:22.481 INFO:teuthology.orchestra.run.smithi112.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-10T03:31:22.481 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T03:31:22.481 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T03:31:22.481 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T03:31:22.481 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T03:31:22.482 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T03:31:22.482 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T03:31:22.482 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T03:31:22.482 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T03:31:22.482 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T03:31:22.482 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T03:31:22.482 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T03:31:22.482 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T03:31:22.482 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T03:31:22.482 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T03:31:22.482 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T03:31:22.482 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T03:31:22.482 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T03:31:22.483 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T03:31:22.483 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T03:31:22.483 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T03:31:22.483 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T03:31:22.483 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T03:31:22.483 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T03:31:22.483 INFO:teuthology.orchestra.run.smithi112.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T03:31:22.483 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T03:31:22.483 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T03:31:22.483 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T03:31:22.483 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T03:31:22.483 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:22.484 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:31:22.791 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-base 2024-09-10T03:31:23.359 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:54 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:31:23.448 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:31:23.490 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:31:23.491 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-10T03:31:23.491 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:31:23.556 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install cephadm 2024-09-10T03:31:23.910 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/4 2024-09-10T03:31:23.910 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/4 2024-09-10T03:31:23.910 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/4 2024-09-10T03:31:23.910 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-10T03:31:24.114 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:55 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:31:24.210 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-10T03:31:24.210 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:24.210 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-10T03:31:24.210 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:24.210 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:24.210 INFO:teuthology.orchestra.run.smithi169.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T03:31:24.210 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T03:31:24.210 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:24.211 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:31:24.246 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:31:24.247 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:24.247 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-10T03:31:24.247 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:24.247 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-10T03:31:24.247 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 765 k 2024-09-10T03:31:24.247 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:24.247 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-10T03:31:24.247 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:24.247 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-09-10T03:31:24.247 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:24.247 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 765 k 2024-09-10T03:31:24.247 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 771 k 2024-09-10T03:31:24.248 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-10T03:31:24.511 INFO:teuthology.orchestra.run.smithi112.stdout:cephadm-19.1.1-298.gb53b6d3e.el9.noarch.rpm 2.8 MB/s | 765 kB 00:00 2024-09-10T03:31:24.511 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:31:24.512 INFO:teuthology.orchestra.run.smithi112.stdout:Total 2.8 MB/s | 765 kB 00:00 2024-09-10T03:31:24.512 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-10T03:31:24.517 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-10T03:31:24.517 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-10T03:31:24.523 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-10T03:31:24.523 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-10T03:31:24.526 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph 2024-09-10T03:31:24.598 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-10T03:31:25.085 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-10T03:31:25.109 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:40 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:31:25.174 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-10T03:31:25.270 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:31:25.273 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:31:25.273 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-10T03:31:25.273 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:31:25.273 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-10T03:31:25.273 INFO:teuthology.orchestra.run.smithi169.stdout: ceph x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 6.5 k 2024-09-10T03:31:25.273 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-10T03:31:25.273 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 2.4 M 2024-09-10T03:31:25.273 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 1.5 M 2024-09-10T03:31:25.273 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 249 k 2024-09-10T03:31:25.273 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 4.7 M 2024-09-10T03:31:25.273 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 17 M 2024-09-10T03:31:25.273 INFO:teuthology.orchestra.run.smithi169.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-10T03:31:25.274 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-10T03:31:25.274 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-10T03:31:25.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T03:31:25.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T03:31:25.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T03:31:25.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T03:31:25.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T03:31:25.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T03:31:25.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T03:31:25.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-10T03:31:25.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T03:31:25.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T03:31:25.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T03:31:25.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T03:31:25.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T03:31:25.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T03:31:25.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T03:31:25.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T03:31:25.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T03:31:25.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T03:31:25.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T03:31:25.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T03:31:25.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T03:31:25.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T03:31:25.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T03:31:25.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T03:31:25.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T03:31:25.276 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T03:31:25.276 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T03:31:25.276 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T03:31:25.276 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:25.276 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-10T03:31:25.276 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:31:25.276 INFO:teuthology.orchestra.run.smithi169.stdout:Install 37 Packages 2024-09-10T03:31:25.276 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:25.277 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 30 M 2024-09-10T03:31:25.277 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 108 M 2024-09-10T03:31:25.277 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-10T03:31:25.506 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-10T03:31:25.733 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-10T03:31:25.733 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:25.733 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-10T03:31:25.734 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-10T03:31:25.734 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:25.734 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:31:25.945 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-immutable-object-cache 2024-09-10T03:31:26.053 INFO:teuthology.orchestra.run.smithi169.stdout:(1/37): ceph-19.1.1-298.gb53b6d3e.el9.x86_64.rp 50 kB/s | 6.5 kB 00:00 2024-09-10T03:31:26.270 INFO:teuthology.orchestra.run.smithi169.stdout:(2/37): ceph-mds-19.1.1-298.gb53b6d3e.el9.x86_6 6.8 MB/s | 2.4 MB 00:00 2024-09-10T03:31:26.312 INFO:teuthology.orchestra.run.smithi169.stdout:(3/37): ceph-mgr-19.1.1-298.gb53b6d3e.el9.x86_6 3.9 MB/s | 1.5 MB 00:00 2024-09-10T03:31:26.404 INFO:teuthology.orchestra.run.smithi169.stdout:(4/37): ceph-mgr-modules-core-19.1.1-298.gb53b6 2.7 MB/s | 249 kB 00:00 2024-09-10T03:31:26.504 INFO:teuthology.orchestra.run.smithi169.stdout:(5/37): ceph-mon-19.1.1-298.gb53b6d3e.el9.x86_6 10 MB/s | 4.7 MB 00:00 2024-09-10T03:31:26.516 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:57 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:31:26.638 INFO:teuthology.orchestra.run.smithi169.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-10T03:31:26.655 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:31:26.656 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:26.656 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-09-10T03:31:26.656 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:26.656 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-10T03:31:26.656 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 145 k 2024-09-10T03:31:26.656 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:26.656 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-10T03:31:26.657 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:26.657 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-09-10T03:31:26.657 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:26.657 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 145 k 2024-09-10T03:31:26.657 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 435 k 2024-09-10T03:31:26.657 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-10T03:31:26.813 INFO:teuthology.orchestra.run.smithi169.stdout:(7/37): ceph-osd-19.1.1-298.gb53b6d3e.el9.x86_6 31 MB/s | 17 MB 00:00 2024-09-10T03:31:26.838 INFO:teuthology.orchestra.run.smithi169.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 532 kB/s | 106 kB 00:00 2024-09-10T03:31:26.858 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-immutable-object-cache-19.1.1-298.gb53b6d3 721 kB/s | 145 kB 00:00 2024-09-10T03:31:26.858 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:31:26.859 INFO:teuthology.orchestra.run.smithi112.stdout:Total 716 kB/s | 145 kB 00:00 2024-09-10T03:31:26.859 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-10T03:31:26.866 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-10T03:31:26.867 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-10T03:31:26.897 INFO:teuthology.orchestra.run.smithi169.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-09-10T03:31:26.923 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-10T03:31:26.923 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-10T03:31:26.939 INFO:teuthology.orchestra.run.smithi169.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.8 MB/s | 1.2 MB 00:00 2024-09-10T03:31:26.964 INFO:teuthology.orchestra.run.smithi169.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-10T03:31:26.989 INFO:teuthology.orchestra.run.smithi169.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 769 kB/s | 135 kB 00:00 2024-09-10T03:31:27.023 INFO:teuthology.orchestra.run.smithi169.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-09-10T03:31:27.048 INFO:teuthology.orchestra.run.smithi169.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 592 kB/s | 35 kB 00:00 2024-09-10T03:31:27.070 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-10T03:31:27.073 INFO:teuthology.orchestra.run.smithi169.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-10T03:31:27.115 INFO:teuthology.orchestra.run.smithi169.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 835 kB/s | 77 kB 00:00 2024-09-10T03:31:27.134 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.e 1/1 2024-09-10T03:31:27.141 INFO:teuthology.orchestra.run.smithi169.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 622 kB/s | 42 kB 00:00 2024-09-10T03:31:27.159 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.e 1/1 2024-09-10T03:31:27.159 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T03:31:27.159 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T03:31:27.160 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:27.166 INFO:teuthology.orchestra.run.smithi169.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 433 kB/s | 51 kB 00:00 2024-09-10T03:31:27.216 INFO:teuthology.orchestra.run.smithi169.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-10T03:31:27.241 INFO:teuthology.orchestra.run.smithi169.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 177 kB/s | 22 kB 00:00 2024-09-10T03:31:27.267 INFO:teuthology.orchestra.run.smithi169.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-10T03:31:27.301 INFO:teuthology.orchestra.run.smithi169.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 409 kB/s | 11 kB 00:00 2024-09-10T03:31:27.326 INFO:teuthology.orchestra.run.smithi169.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-10T03:31:27.352 INFO:teuthology.orchestra.run.smithi169.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 365 kB/s | 18 kB 00:00 2024-09-10T03:31:27.376 INFO:teuthology.orchestra.run.smithi169.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-10T03:31:27.401 INFO:teuthology.orchestra.run.smithi169.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-10T03:31:27.426 INFO:teuthology.orchestra.run.smithi169.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-10T03:31:27.452 INFO:teuthology.orchestra.run.smithi169.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-09-10T03:31:27.477 INFO:teuthology.orchestra.run.smithi169.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-10T03:31:27.502 INFO:teuthology.orchestra.run.smithi169.stdout:(30/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-10T03:31:27.528 INFO:teuthology.orchestra.run.smithi169.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 325 kB/s | 16 kB 00:00 2024-09-10T03:31:27.556 INFO:teuthology.orchestra.run.smithi169.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-10T03:31:27.579 INFO:teuthology.orchestra.run.smithi169.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-09-10T03:31:27.604 INFO:teuthology.orchestra.run.smithi169.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-10T03:31:27.629 INFO:teuthology.orchestra.run.smithi169.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 798 kB/s | 20 kB 00:00 2024-09-10T03:31:27.654 INFO:teuthology.orchestra.run.smithi169.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-10T03:31:27.688 INFO:teuthology.orchestra.run.smithi169.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-10T03:31:27.692 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:31:27.693 INFO:teuthology.orchestra.run.smithi169.stdout:Total 13 MB/s | 30 MB 00:02 2024-09-10T03:31:27.844 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-10T03:31:27.885 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-10T03:31:27.885 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-10T03:31:28.067 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.e 1/1 2024-09-10T03:31:28.067 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:28.067 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-10T03:31:28.067 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:28.067 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:28.067 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:31:28.327 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr 2024-09-10T03:31:28.346 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-10T03:31:28.346 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-10T03:31:28.901 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:59 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:31:28.992 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:31:29.034 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:31:29.035 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-10T03:31:29.035 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:31:29.108 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-dashboard 2024-09-10T03:31:29.663 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:00 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:31:29.693 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-10T03:31:29.766 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-10T03:31:29.796 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:31:29.797 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:29.797 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-10T03:31:29.797 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:29.797 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-10T03:31:29.797 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 3.6 M 2024-09-10T03:31:29.797 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-10T03:31:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-10T03:31:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 32 k 2024-09-10T03:31:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 16 k 2024-09-10T03:31:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-10T03:31:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-10T03:31:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-10T03:31:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-10T03:31:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-10T03:31:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-10T03:31:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T03:31:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T03:31:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:29.798 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-10T03:31:29.799 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:29.799 INFO:teuthology.orchestra.run.smithi112.stdout:Install 12 Packages 2024-09-10T03:31:29.799 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:29.799 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-10T03:31:29.800 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 8.7 M 2024-09-10T03:31:29.800 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 104 M 2024-09-10T03:31:29.800 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-10T03:31:29.843 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-10T03:31:29.918 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-10T03:31:29.962 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-10T03:31:29.995 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-10T03:31:30.096 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-10T03:31:30.182 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-10T03:31:30.223 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-10T03:31:30.267 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-10T03:31:30.304 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-10T03:31:30.339 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-10T03:31:30.381 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-10T03:31:30.421 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-10T03:31:30.429 INFO:teuthology.orchestra.run.smithi112.stdout:(1/12): ceph-grafana-dashboards-19.1.1-298.gb53 195 kB/s | 32 kB 00:00 2024-09-10T03:31:30.463 INFO:teuthology.orchestra.run.smithi112.stdout:(2/12): ceph-prometheus-alerts-19.1.1-298.gb53b 83 kB/s | 16 kB 00:00 2024-09-10T03:31:30.473 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-10T03:31:30.507 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-10T03:31:30.556 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-10T03:31:30.580 INFO:teuthology.orchestra.run.smithi112.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.3 MB/s | 269 kB 00:00 2024-09-10T03:31:30.591 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-10T03:31:30.671 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-10T03:31:30.672 INFO:teuthology.orchestra.run.smithi112.stdout:(4/12): ceph-mgr-dashboard-19.1.1-298.gb53b6d3e 8.8 MB/s | 3.6 MB 00:00 2024-09-10T03:31:30.705 INFO:teuthology.orchestra.run.smithi112.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.7 MB/s | 1.0 MB 00:00 2024-09-10T03:31:30.747 INFO:teuthology.orchestra.run.smithi112.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.0 MB/s | 863 kB 00:00 2024-09-10T03:31:30.781 INFO:teuthology.orchestra.run.smithi112.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 258 kB/s | 19 kB 00:00 2024-09-10T03:31:30.809 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-10T03:31:30.823 INFO:teuthology.orchestra.run.smithi112.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.6 MB/s | 548 kB 00:00 2024-09-10T03:31:30.848 INFO:teuthology.orchestra.run.smithi112.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-10T03:31:30.873 INFO:teuthology.orchestra.run.smithi112.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.6 MB/s | 144 kB 00:00 2024-09-10T03:31:30.889 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-10T03:31:30.898 INFO:teuthology.orchestra.run.smithi112.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-10T03:31:30.949 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-10T03:31:30.965 INFO:teuthology.orchestra.run.smithi112.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.4 MB/s | 2.0 MB 00:00 2024-09-10T03:31:30.970 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:31:30.970 INFO:teuthology.orchestra.run.smithi112.stdout:Total 7.4 MB/s | 8.7 MB 00:01 2024-09-10T03:31:30.996 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-10T03:31:31.017 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-10T03:31:31.033 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-10T03:31:31.033 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-10T03:31:31.049 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-10T03:31:31.109 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-10T03:31:31.221 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-10T03:31:31.222 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-10T03:31:31.305 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-10T03:31:31.358 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-10T03:31:31.447 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-10T03:31:31.544 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-10T03:31:31.584 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-10T03:31:31.701 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-10T03:31:31.745 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-10T03:31:31.816 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-10T03:31:31.851 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-10T03:31:31.864 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.n 32/37 2024-09-10T03:31:31.894 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 33/37 2024-09-10T03:31:31.897 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-10T03:31:31.921 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 33/37 2024-09-10T03:31:31.921 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T03:31:31.921 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T03:31:31.921 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T03:31:31.921 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T03:31:31.921 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:31.928 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-10T03:31:32.028 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-10T03:31:32.153 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-10T03:31:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-10T03:31:32.284 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-10T03:31:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-10T03:31:32.351 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9. 10/12 2024-09-10T03:31:33.370 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 34/37 2024-09-10T03:31:33.398 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 34/37 2024-09-10T03:31:33.398 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T03:31:33.398 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T03:31:33.398 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T03:31:33.398 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T03:31:33.398 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:33.401 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9 11/12 2024-09-10T03:31:33.435 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noar 12/12 2024-09-10T03:31:33.846 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 35/37 2024-09-10T03:31:33.870 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 35/37 2024-09-10T03:31:33.870 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T03:31:33.870 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T03:31:33.870 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T03:31:33.870 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T03:31:33.870 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:34.096 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 36/37 2024-09-10T03:31:34.120 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 36/37 2024-09-10T03:31:34.120 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T03:31:34.121 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T03:31:34.121 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T03:31:34.121 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T03:31:34.121 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:34.193 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 37/37 2024-09-10T03:31:34.783 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noar 12/12 2024-09-10T03:31:34.784 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9 1/12 2024-09-10T03:31:34.784 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noar 2/12 2024-09-10T03:31:34.784 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9. 3/12 2024-09-10T03:31:34.784 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-10T03:31:34.784 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-10T03:31:34.784 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-10T03:31:34.784 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-10T03:31:34.784 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-10T03:31:34.784 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-10T03:31:34.784 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-10T03:31:34.784 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-10T03:31:35.099 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-10T03:31:35.099 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:35.099 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-10T03:31:35.099 INFO:teuthology.orchestra.run.smithi112.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-10T03:31:35.099 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-10T03:31:35.099 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-10T03:31:35.099 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-10T03:31:35.100 INFO:teuthology.orchestra.run.smithi112.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-10T03:31:35.100 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-10T03:31:35.100 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-10T03:31:35.100 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-10T03:31:35.100 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-10T03:31:35.100 INFO:teuthology.orchestra.run.smithi112.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-10T03:31:35.100 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T03:31:35.100 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T03:31:35.100 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:35.100 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:31:35.386 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-10T03:31:35.941 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:06 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:31:36.009 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 37/37 2024-09-10T03:31:36.009 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/37 2024-09-10T03:31:36.009 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/37 2024-09-10T03:31:36.010 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/37 2024-09-10T03:31:36.010 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/37 2024-09-10T03:31:36.010 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 5/37 2024-09-10T03:31:36.010 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.n 6/37 2024-09-10T03:31:36.010 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-10T03:31:36.010 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-10T03:31:36.010 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-10T03:31:36.010 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-10T03:31:36.010 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-10T03:31:36.010 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-10T03:31:36.010 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-10T03:31:36.010 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-10T03:31:36.010 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-10T03:31:36.010 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-10T03:31:36.011 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-10T03:31:36.011 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-10T03:31:36.011 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-10T03:31:36.011 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-10T03:31:36.011 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-10T03:31:36.011 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-10T03:31:36.011 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-10T03:31:36.011 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-10T03:31:36.011 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-10T03:31:36.011 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-10T03:31:36.011 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-10T03:31:36.012 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-10T03:31:36.012 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-10T03:31:36.013 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-10T03:31:36.013 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-10T03:31:36.013 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-10T03:31:36.013 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-10T03:31:36.013 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-10T03:31:36.013 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-10T03:31:36.013 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-10T03:31:36.073 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:31:36.074 INFO:teuthology.orchestra.run.smithi112.stdout:====================================================================================== 2024-09-10T03:31:36.074 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-10T03:31:36.074 INFO:teuthology.orchestra.run.smithi112.stdout:====================================================================================== 2024-09-10T03:31:36.074 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-10T03:31:36.074 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 7.4 M 2024-09-10T03:31:36.074 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-10T03:31:36.074 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T03:31:36.074 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T03:31:36.074 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T03:31:36.075 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T03:31:36.075 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T03:31:36.075 INFO:teuthology.orchestra.run.smithi112.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T03:31:36.075 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T03:31:36.075 INFO:teuthology.orchestra.run.smithi112.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T03:31:36.075 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T03:31:36.075 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T03:31:36.075 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T03:31:36.075 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:36.075 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-10T03:31:36.075 INFO:teuthology.orchestra.run.smithi112.stdout:====================================================================================== 2024-09-10T03:31:36.075 INFO:teuthology.orchestra.run.smithi112.stdout:Install 12 Packages 2024-09-10T03:31:36.075 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:36.076 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 38 M 2024-09-10T03:31:36.076 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 212 M 2024-09-10T03:31:36.076 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-10T03:31:36.356 INFO:teuthology.orchestra.run.smithi112.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.2 MB/s | 191 kB 00:00 2024-09-10T03:31:36.415 INFO:teuthology.orchestra.run.smithi112.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.7 MB/s | 801 kB 00:00 2024-09-10T03:31:36.673 INFO:teuthology.orchestra.run.smithi112.stdout:(3/12): ceph-mgr-diskprediction-local-19.1.1-29 16 MB/s | 7.4 MB 00:00 2024-09-10T03:31:36.909 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-10T03:31:36.909 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:36.909 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-10T03:31:36.909 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:36.909 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:36.909 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:36.909 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-10T03:31:36.909 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:36.909 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:36.909 INFO:teuthology.orchestra.run.smithi169.stdout: lua-5.4.4-4.el9.x86_64 2024-09-10T03:31:36.909 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-10T03:31:36.910 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-10T03:31:36.910 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T03:31:36.910 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T03:31:36.910 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T03:31:36.910 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T03:31:36.910 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T03:31:36.910 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T03:31:36.910 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T03:31:36.910 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T03:31:36.910 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T03:31:36.910 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T03:31:36.910 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T03:31:36.910 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T03:31:36.910 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T03:31:36.911 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T03:31:36.911 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T03:31:36.911 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T03:31:36.911 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T03:31:36.911 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T03:31:36.911 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T03:31:36.911 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T03:31:36.911 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T03:31:36.911 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T03:31:36.911 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T03:31:36.911 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T03:31:36.911 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T03:31:36.911 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T03:31:36.911 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T03:31:36.912 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T03:31:36.912 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:36.912 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:31:36.932 INFO:teuthology.orchestra.run.smithi112.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 57 kB/s | 33 kB 00:00 2024-09-10T03:31:37.182 INFO:teuthology.orchestra.run.smithi112.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 36 kB/s | 18 kB 00:00 2024-09-10T03:31:37.251 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-base 2024-09-10T03:31:37.541 INFO:teuthology.orchestra.run.smithi112.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 63 kB/s | 38 kB 00:00 2024-09-10T03:31:37.821 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:52 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:31:37.911 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:31:37.916 INFO:teuthology.orchestra.run.smithi112.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.0 MB/s | 3.0 MB 00:01 2024-09-10T03:31:37.953 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:31:37.954 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-10T03:31:37.954 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:31:38.015 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install cephadm 2024-09-10T03:31:38.383 INFO:teuthology.orchestra.run.smithi112.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 291 kB/s | 245 kB 00:00 2024-09-10T03:31:38.580 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:53 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:31:38.711 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:31:38.712 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:31:38.712 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-10T03:31:38.712 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:31:38.712 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-10T03:31:38.713 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 765 k 2024-09-10T03:31:38.713 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:38.713 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-10T03:31:38.713 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:31:38.713 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-09-10T03:31:38.713 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:38.713 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 765 k 2024-09-10T03:31:38.713 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 771 k 2024-09-10T03:31:38.713 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-10T03:31:38.958 INFO:teuthology.orchestra.run.smithi112.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 2.8 MB/s | 4.9 MB 00:01 2024-09-10T03:31:38.998 INFO:teuthology.orchestra.run.smithi169.stdout:cephadm-19.1.1-298.gb53b6d3e.el9.noarch.rpm 2.6 MB/s | 765 kB 00:00 2024-09-10T03:31:38.998 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:31:38.999 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.6 MB/s | 765 kB 00:00 2024-09-10T03:31:38.999 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-10T03:31:39.004 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-10T03:31:39.004 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-10T03:31:39.010 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-10T03:31:39.010 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-10T03:31:39.033 INFO:teuthology.orchestra.run.smithi112.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.6 MB/s | 5.1 MB 00:01 2024-09-10T03:31:39.082 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-10T03:31:39.258 INFO:teuthology.orchestra.run.smithi112.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 286 kB/s | 250 kB 00:00 2024-09-10T03:31:39.521 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-10T03:31:39.609 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-10T03:31:40.003 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-10T03:31:40.242 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-10T03:31:40.242 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:40.242 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-10T03:31:40.242 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-10T03:31:40.242 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:40.242 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:31:40.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-immutable-object-cache 2024-09-10T03:31:41.020 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:56 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:31:41.157 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:31:41.158 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:31:41.158 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-09-10T03:31:41.158 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:31:41.158 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-10T03:31:41.158 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 145 k 2024-09-10T03:31:41.158 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:41.158 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-10T03:31:41.158 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:31:41.158 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-09-10T03:31:41.158 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:41.158 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 145 k 2024-09-10T03:31:41.159 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 435 k 2024-09-10T03:31:41.159 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-10T03:31:41.370 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-immutable-object-cache-19.1.1-298.gb53b6d3 686 kB/s | 145 kB 00:00 2024-09-10T03:31:41.371 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:31:41.371 INFO:teuthology.orchestra.run.smithi169.stdout:Total 681 kB/s | 145 kB 00:00 2024-09-10T03:31:41.371 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-10T03:31:41.378 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-10T03:31:41.378 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-10T03:31:41.437 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-10T03:31:41.437 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-10T03:31:41.596 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-10T03:31:41.688 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.e 1/1 2024-09-10T03:31:41.713 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.e 1/1 2024-09-10T03:31:41.713 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T03:31:41.713 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T03:31:41.713 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.e 1/1 2024-09-10T03:31:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:42.622 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-10T03:31:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:31:42.622 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:42.623 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:31:42.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr 2024-09-10T03:31:43.422 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:58 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:31:43.517 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:31:43.564 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:31:43.565 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-10T03:31:43.565 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:31:43.642 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-dashboard 2024-09-10T03:31:44.206 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:59 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:31:44.300 INFO:teuthology.orchestra.run.smithi112.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.0 MB/s | 16 MB 00:05 2024-09-10T03:31:44.301 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:31:44.301 INFO:teuthology.orchestra.run.smithi112.stdout:Total 4.6 MB/s | 38 MB 00:08 2024-09-10T03:31:44.342 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:31:44.343 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:31:44.343 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-10T03:31:44.343 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:31:44.343 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-10T03:31:44.343 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 3.6 M 2024-09-10T03:31:44.343 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-10T03:31:44.344 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-10T03:31:44.344 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 32 k 2024-09-10T03:31:44.344 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 16 k 2024-09-10T03:31:44.344 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-10T03:31:44.344 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-10T03:31:44.344 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-10T03:31:44.344 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-10T03:31:44.344 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-10T03:31:44.344 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-10T03:31:44.344 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T03:31:44.344 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T03:31:44.344 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:44.345 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-10T03:31:44.345 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:31:44.345 INFO:teuthology.orchestra.run.smithi169.stdout:Install 12 Packages 2024-09-10T03:31:44.345 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:44.345 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 8.7 M 2024-09-10T03:31:44.345 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 104 M 2024-09-10T03:31:44.346 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-10T03:31:44.632 INFO:teuthology.orchestra.run.smithi169.stdout:(1/12): ceph-grafana-dashboards-19.1.1-298.gb53 203 kB/s | 32 kB 00:00 2024-09-10T03:31:44.657 INFO:teuthology.orchestra.run.smithi169.stdout:(2/12): ceph-prometheus-alerts-19.1.1-298.gb53b 90 kB/s | 16 kB 00:00 2024-09-10T03:31:44.659 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-10T03:31:44.686 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-10T03:31:44.687 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-10T03:31:44.866 INFO:teuthology.orchestra.run.smithi169.stdout:(3/12): ceph-mgr-dashboard-19.1.1-298.gb53b6d3e 9.1 MB/s | 3.6 MB 00:00 2024-09-10T03:31:44.891 INFO:teuthology.orchestra.run.smithi169.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 269 kB 00:00 2024-09-10T03:31:44.975 INFO:teuthology.orchestra.run.smithi169.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 6.4 MB/s | 548 kB 00:00 2024-09-10T03:31:45.000 INFO:teuthology.orchestra.run.smithi169.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 783 kB/s | 19 kB 00:00 2024-09-10T03:31:45.034 INFO:teuthology.orchestra.run.smithi169.stdout:(7/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.5 MB/s | 1.0 MB 00:00 2024-09-10T03:31:45.092 INFO:teuthology.orchestra.run.smithi169.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-09-10T03:31:45.118 INFO:teuthology.orchestra.run.smithi169.stdout:(9/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.4 MB/s | 863 kB 00:00 2024-09-10T03:31:45.118 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-10T03:31:45.118 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-10T03:31:45.134 INFO:teuthology.orchestra.run.smithi169.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 737 kB/s | 31 kB 00:00 2024-09-10T03:31:45.193 INFO:teuthology.orchestra.run.smithi169.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-09-10T03:31:45.243 INFO:teuthology.orchestra.run.smithi169.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 8.4 MB/s | 2.0 MB 00:00 2024-09-10T03:31:45.247 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:31:45.247 INFO:teuthology.orchestra.run.smithi169.stdout:Total 9.6 MB/s | 8.7 MB 00:00 2024-09-10T03:31:45.296 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-10T03:31:45.314 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-10T03:31:45.314 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-10T03:31:45.507 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-10T03:31:45.507 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-10T03:31:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-10T03:31:45.957 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-10T03:31:45.998 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-10T03:31:46.032 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-10T03:31:46.097 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-10T03:31:46.119 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-10T03:31:46.141 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-10T03:31:46.154 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-10T03:31:46.201 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-10T03:31:46.239 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-10T03:31:46.325 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-10T03:31:46.346 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-10T03:31:46.365 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-10T03:31:46.442 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-10T03:31:46.482 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-10T03:31:46.575 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-10T03:31:46.627 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-10T03:31:46.656 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-10T03:31:46.688 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9. 10/12 2024-09-10T03:31:47.093 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-10T03:31:47.219 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-10T03:31:47.750 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9 11/12 2024-09-10T03:31:47.797 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noar 12/12 2024-09-10T03:31:48.443 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T03:31:49.249 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noar 12/12 2024-09-10T03:31:49.249 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9 1/12 2024-09-10T03:31:49.249 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noar 2/12 2024-09-10T03:31:49.249 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9. 3/12 2024-09-10T03:31:49.250 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-10T03:31:49.250 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-10T03:31:49.250 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-10T03:31:49.250 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-10T03:31:49.250 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-10T03:31:49.251 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-10T03:31:49.251 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-10T03:31:49.251 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-10T03:31:49.577 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-10T03:31:49.577 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:49.577 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-10T03:31:49.577 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-10T03:31:49.577 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-10T03:31:49.578 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-10T03:31:49.578 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-10T03:31:49.578 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-10T03:31:49.578 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-10T03:31:49.578 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-10T03:31:49.579 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-10T03:31:49.579 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-10T03:31:49.579 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-10T03:31:49.579 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T03:31:49.579 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T03:31:49.579 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:49.580 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:31:49.662 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-10T03:31:49.688 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d 12/12 2024-09-10T03:31:49.822 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-10T03:31:50.386 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:05 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:31:50.521 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:31:50.522 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-09-10T03:31:50.522 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-10T03:31:50.522 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-09-10T03:31:50.523 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-10T03:31:50.523 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 7.4 M 2024-09-10T03:31:50.523 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-10T03:31:50.523 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T03:31:50.523 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T03:31:50.523 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T03:31:50.523 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T03:31:50.523 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T03:31:50.523 INFO:teuthology.orchestra.run.smithi169.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T03:31:50.523 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T03:31:50.524 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T03:31:50.524 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T03:31:50.524 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T03:31:50.524 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T03:31:50.524 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:50.524 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-10T03:31:50.524 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-09-10T03:31:50.524 INFO:teuthology.orchestra.run.smithi169.stdout:Install 12 Packages 2024-09-10T03:31:50.524 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:31:50.525 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 38 M 2024-09-10T03:31:50.525 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 212 M 2024-09-10T03:31:50.525 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-10T03:31:51.035 INFO:teuthology.orchestra.run.smithi169.stdout:(1/12): ceph-mgr-diskprediction-local-19.1.1-29 17 MB/s | 7.4 MB 00:00 2024-09-10T03:31:51.093 INFO:teuthology.orchestra.run.smithi169.stdout:(2/12): libquadmath-11.5.0-2.el9.x86_64.rpm 380 kB/s | 191 kB 00:00 2024-09-10T03:31:51.143 INFO:teuthology.orchestra.run.smithi169.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 302 kB/s | 33 kB 00:00 2024-09-10T03:31:51.194 INFO:teuthology.orchestra.run.smithi169.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-10T03:31:51.244 INFO:teuthology.orchestra.run.smithi169.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 768 kB/s | 38 kB 00:00 2024-09-10T03:31:51.303 INFO:teuthology.orchestra.run.smithi169.stdout:(6/12): libgfortran-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 801 kB 00:00 2024-09-10T03:31:51.503 INFO:teuthology.orchestra.run.smithi169.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-09-10T03:31:51.670 INFO:teuthology.orchestra.run.smithi169.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.2 MB/s | 3.0 MB 00:00 2024-09-10T03:31:51.728 INFO:teuthology.orchestra.run.smithi169.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-09-10T03:31:51.912 INFO:teuthology.orchestra.run.smithi169.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 7.4 MB/s | 4.9 MB 00:00 2024-09-10T03:31:52.221 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d 12/12 2024-09-10T03:31:52.221 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d 1/12 2024-09-10T03:31:52.221 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-10T03:31:52.221 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-10T03:31:52.221 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-10T03:31:52.221 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-10T03:31:52.221 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-10T03:31:52.221 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-10T03:31:52.221 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-10T03:31:52.221 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-10T03:31:52.221 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T03:31:52.222 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-10T03:31:52.287 INFO:teuthology.orchestra.run.smithi169.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.5 MB/s | 5.1 MB 00:00 2024-09-10T03:31:52.605 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-10T03:31:52.605 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:52.605 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-10T03:31:52.605 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-10T03:31:52.605 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T03:31:52.605 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T03:31:52.605 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T03:31:52.605 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T03:31:52.606 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T03:31:52.606 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T03:31:52.606 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T03:31:52.606 INFO:teuthology.orchestra.run.smithi112.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T03:31:52.606 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T03:31:52.606 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T03:31:52.606 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T03:31:52.606 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:52.606 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:31:53.026 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-rook 2024-09-10T03:31:53.520 INFO:teuthology.orchestra.run.smithi169.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.1 MB/s | 16 MB 00:01 2024-09-10T03:31:53.521 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:31:53.522 INFO:teuthology.orchestra.run.smithi169.stdout:Total 13 MB/s | 38 MB 00:02 2024-09-10T03:31:53.603 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:24 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:31:53.737 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:31:53.740 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:53.740 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-10T03:31:53.740 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:53.740 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-10T03:31:53.741 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 49 k 2024-09-10T03:31:53.741 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-10T03:31:53.741 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T03:31:53.741 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T03:31:53.741 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T03:31:53.741 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T03:31:53.741 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T03:31:53.741 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T03:31:53.741 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T03:31:53.741 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T03:31:53.741 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T03:31:53.741 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T03:31:53.741 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T03:31:53.742 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T03:31:53.742 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:53.742 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-10T03:31:53.742 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:53.742 INFO:teuthology.orchestra.run.smithi112.stdout:Install 13 Packages 2024-09-10T03:31:53.742 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:53.742 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 2.2 M 2024-09-10T03:31:53.742 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 26 M 2024-09-10T03:31:53.742 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-10T03:31:53.878 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-10T03:31:53.903 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-10T03:31:53.903 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-10T03:31:54.013 INFO:teuthology.orchestra.run.smithi112.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 136 kB/s | 19 kB 00:00 2024-09-10T03:31:54.038 INFO:teuthology.orchestra.run.smithi112.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 154 kB/s | 26 kB 00:00 2024-09-10T03:31:54.063 INFO:teuthology.orchestra.run.smithi112.stdout:(3/13): ceph-mgr-rook-19.1.1-298.gb53b6d3e.el9. 257 kB/s | 49 kB 00:00 2024-09-10T03:31:54.189 INFO:teuthology.orchestra.run.smithi112.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-09-10T03:31:54.214 INFO:teuthology.orchestra.run.smithi112.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.1 MB/s | 222 kB 00:00 2024-09-10T03:31:54.247 INFO:teuthology.orchestra.run.smithi112.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 917 kB/s | 54 kB 00:00 2024-09-10T03:31:54.281 INFO:teuthology.orchestra.run.smithi112.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 479 kB/s | 32 kB 00:00 2024-09-10T03:31:54.306 INFO:teuthology.orchestra.run.smithi112.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 247 kB/s | 14 kB 00:00 2024-09-10T03:31:54.328 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-10T03:31:54.328 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-10T03:31:54.332 INFO:teuthology.orchestra.run.smithi112.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.0 MB/s | 279 kB 00:00 2024-09-10T03:31:54.357 INFO:teuthology.orchestra.run.smithi112.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.9 MB/s | 220 kB 00:00 2024-09-10T03:31:54.399 INFO:teuthology.orchestra.run.smithi112.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 11 MB/s | 1.0 MB 00:00 2024-09-10T03:31:54.424 INFO:teuthology.orchestra.run.smithi112.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-09-10T03:31:54.449 INFO:teuthology.orchestra.run.smithi112.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 504 kB/s | 59 kB 00:00 2024-09-10T03:31:54.452 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:31:54.452 INFO:teuthology.orchestra.run.smithi112.stdout:Total 3.1 MB/s | 2.2 MB 00:00 2024-09-10T03:31:54.514 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-10T03:31:54.529 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-10T03:31:54.530 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-10T03:31:54.617 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-10T03:31:54.619 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-10T03:31:55.101 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-10T03:31:55.132 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-10T03:31:55.174 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-10T03:31:55.219 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-10T03:31:55.305 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-10T03:31:55.318 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-10T03:31:55.362 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-10T03:31:55.367 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-10T03:31:55.430 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-10T03:31:55.468 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-10T03:31:55.509 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-10T03:31:55.547 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-10T03:31:55.585 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-10T03:31:55.619 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-10T03:31:55.662 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-10T03:31:55.721 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-10T03:31:55.777 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-10T03:31:56.317 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-10T03:31:56.440 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-10T03:31:56.491 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-10T03:31:56.541 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-10T03:31:56.575 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-10T03:31:56.623 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-10T03:31:56.645 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 13/13 2024-09-10T03:31:57.422 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 13/13 2024-09-10T03:31:57.422 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 1/13 2024-09-10T03:31:57.422 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-10T03:31:57.422 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-10T03:31:57.422 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-10T03:31:57.423 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-10T03:31:57.423 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-10T03:31:57.423 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-10T03:31:57.423 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-10T03:31:57.423 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-10T03:31:57.423 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-10T03:31:57.423 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-10T03:31:57.423 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-10T03:31:57.672 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T03:31:57.753 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-10T03:31:57.754 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:57.754 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-10T03:31:57.754 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-10T03:31:57.754 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T03:31:57.754 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T03:31:57.754 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T03:31:57.754 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T03:31:57.754 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T03:31:57.754 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T03:31:57.754 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T03:31:57.754 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T03:31:57.754 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T03:31:57.755 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T03:31:57.755 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T03:31:57.755 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T03:31:57.755 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:57.755 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:31:58.019 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-cephadm 2024-09-10T03:31:58.583 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:29 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:31:58.707 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-10T03:31:58.716 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:31:58.717 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:58.717 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-10T03:31:58.717 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:58.717 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-10T03:31:58.717 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 146 k 2024-09-10T03:31:58.717 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-10T03:31:58.718 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T03:31:58.718 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T03:31:58.718 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T03:31:58.718 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T03:31:58.718 INFO:teuthology.orchestra.run.smithi112.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T03:31:58.718 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:58.718 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-10T03:31:58.718 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:31:58.718 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2024-09-10T03:31:58.718 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:31:58.719 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 7.0 M 2024-09-10T03:31:58.719 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 33 M 2024-09-10T03:31:58.719 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-10T03:31:58.735 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d 12/12 2024-09-10T03:31:59.109 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): ceph-mgr-cephadm-19.1.1-298.gb53b6d3e.el 683 kB/s | 146 kB 00:00 2024-09-10T03:31:59.134 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-09-10T03:31:59.176 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.3 MB/s | 58 kB 00:00 2024-09-10T03:31:59.201 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.2 MB/s | 74 kB 00:00 2024-09-10T03:31:59.235 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.3 MB/s | 548 kB 00:00 2024-09-10T03:31:59.452 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-09-10T03:31:59.453 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:31:59.454 INFO:teuthology.orchestra.run.smithi112.stdout:Total 9.5 MB/s | 7.0 MB 00:00 2024-09-10T03:31:59.524 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-10T03:31:59.535 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-10T03:31:59.535 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-10T03:31:59.633 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-10T03:31:59.634 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-10T03:31:59.942 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-10T03:32:00.108 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-10T03:32:00.153 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-10T03:32:00.666 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-10T03:32:00.755 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-10T03:32:00.820 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-10T03:32:00.849 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 6/6 2024-09-10T03:32:01.459 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d 12/12 2024-09-10T03:32:01.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d 1/12 2024-09-10T03:32:01.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-10T03:32:01.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-10T03:32:01.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-10T03:32:01.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-10T03:32:01.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-10T03:32:01.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-10T03:32:01.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-10T03:32:01.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-10T03:32:01.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T03:32:01.460 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-10T03:32:01.655 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 6/6 2024-09-10T03:32:01.656 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/6 2024-09-10T03:32:01.656 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-10T03:32:01.656 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-10T03:32:01.656 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-10T03:32:01.656 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-10T03:32:01.901 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-10T03:32:01.902 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:01.902 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-10T03:32:01.902 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-10T03:32:01.902 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T03:32:01.902 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T03:32:01.902 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T03:32:01.902 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T03:32:01.902 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T03:32:01.902 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T03:32:01.903 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T03:32:01.903 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T03:32:01.903 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T03:32:01.903 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T03:32:01.903 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T03:32:01.903 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:01.903 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:32:01.919 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-10T03:32:01.919 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:01.919 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-10T03:32:01.919 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-10T03:32:01.919 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T03:32:01.919 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T03:32:01.919 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T03:32:01.919 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T03:32:01.919 INFO:teuthology.orchestra.run.smithi112.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T03:32:01.920 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:01.920 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:32:02.182 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-fuse 2024-09-10T03:32:02.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-rook 2024-09-10T03:32:02.750 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:33 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:32:02.857 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:17 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:32:02.884 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:32:02.884 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:02.884 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-10T03:32:02.885 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:02.885 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-10T03:32:02.885 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 897 k 2024-09-10T03:32:02.885 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-10T03:32:02.885 INFO:teuthology.orchestra.run.smithi112.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T03:32:02.885 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:02.885 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-10T03:32:02.885 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:02.885 INFO:teuthology.orchestra.run.smithi112.stdout:Install 2 Packages 2024-09-10T03:32:02.885 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:02.886 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 977 k 2024-09-10T03:32:02.886 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 2.9 M 2024-09-10T03:32:02.886 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-10T03:32:02.993 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:32:02.994 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:02.995 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-10T03:32:02.995 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:02.995 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-10T03:32:02.995 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 49 k 2024-09-10T03:32:02.995 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-10T03:32:02.995 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T03:32:02.995 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T03:32:02.995 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T03:32:02.995 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T03:32:02.995 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T03:32:02.995 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T03:32:02.995 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T03:32:02.995 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T03:32:02.996 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T03:32:02.996 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T03:32:02.996 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T03:32:02.996 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T03:32:02.996 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:02.996 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-10T03:32:02.996 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:02.996 INFO:teuthology.orchestra.run.smithi169.stdout:Install 13 Packages 2024-09-10T03:32:02.996 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:02.997 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 2.2 M 2024-09-10T03:32:02.997 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 26 M 2024-09-10T03:32:02.997 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-10T03:32:03.020 INFO:teuthology.orchestra.run.smithi112.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 710 kB/s | 80 kB 00:00 2024-09-10T03:32:03.203 INFO:teuthology.orchestra.run.smithi112.stdout:(2/2): ceph-fuse-19.1.1-298.gb53b6d3e.el9.x86_6 3.0 MB/s | 897 kB 00:00 2024-09-10T03:32:03.204 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:32:03.204 INFO:teuthology.orchestra.run.smithi112.stdout:Total 3.0 MB/s | 977 kB 00:00 2024-09-10T03:32:03.216 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-10T03:32:03.237 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-10T03:32:03.238 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-10T03:32:03.321 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-10T03:32:03.321 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-10T03:32:03.504 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-10T03:32:03.586 INFO:teuthology.orchestra.run.smithi169.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 329 kB/s | 26 kB 00:00 2024-09-10T03:32:03.611 INFO:teuthology.orchestra.run.smithi169.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 186 kB/s | 19 kB 00:00 2024-09-10T03:32:03.638 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-10T03:32:03.686 INFO:teuthology.orchestra.run.smithi169.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-09-10T03:32:03.702 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/2 2024-09-10T03:32:03.719 INFO:teuthology.orchestra.run.smithi169.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-10T03:32:03.745 INFO:teuthology.orchestra.run.smithi169.stdout:(5/13): ceph-mgr-rook-19.1.1-298.gb53b6d3e.el9. 208 kB/s | 49 kB 00:00 2024-09-10T03:32:03.772 INFO:teuthology.orchestra.run.smithi169.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-10T03:32:03.804 INFO:teuthology.orchestra.run.smithi169.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.3 MB/s | 279 kB 00:00 2024-09-10T03:32:03.854 INFO:teuthology.orchestra.run.smithi169.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 296 kB/s | 32 kB 00:00 2024-09-10T03:32:03.879 INFO:teuthology.orchestra.run.smithi169.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 129 kB/s | 14 kB 00:00 2024-09-10T03:32:03.905 INFO:teuthology.orchestra.run.smithi169.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-10T03:32:03.930 INFO:teuthology.orchestra.run.smithi169.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-10T03:32:03.980 INFO:teuthology.orchestra.run.smithi169.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-09-10T03:32:04.022 INFO:teuthology.orchestra.run.smithi169.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-09-10T03:32:04.025 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:32:04.025 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.2 MB/s | 2.2 MB 00:01 2024-09-10T03:32:04.094 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-10T03:32:04.109 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-10T03:32:04.109 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-10T03:32:04.198 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-10T03:32:04.200 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-10T03:32:04.366 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/2 2024-09-10T03:32:04.366 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/2 2024-09-10T03:32:04.625 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-10T03:32:04.625 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:04.625 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-10T03:32:04.625 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-10T03:32:04.625 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:04.625 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:32:04.704 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-10T03:32:04.867 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-10T03:32:04.906 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados-devel 2024-09-10T03:32:04.931 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-10T03:32:04.995 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-10T03:32:05.033 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-10T03:32:05.074 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-10T03:32:05.183 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-10T03:32:05.295 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-10T03:32:05.350 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-10T03:32:05.483 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:36 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:32:05.621 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:32:05.622 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:05.622 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-09-10T03:32:05.622 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:05.622 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-10T03:32:05.622 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 127 k 2024-09-10T03:32:05.622 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:05.623 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-10T03:32:05.623 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:05.623 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-09-10T03:32:05.623 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:05.623 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 127 k 2024-09-10T03:32:05.623 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 456 k 2024-09-10T03:32:05.623 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-10T03:32:05.832 INFO:teuthology.orchestra.run.smithi112.stdout:librados-devel-19.1.1-298.gb53b6d3e.el9.x86_64. 604 kB/s | 127 kB 00:00 2024-09-10T03:32:05.833 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:32:05.833 INFO:teuthology.orchestra.run.smithi112.stdout:Total 599 kB/s | 127 kB 00:00 2024-09-10T03:32:05.833 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-10T03:32:05.839 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-10T03:32:05.839 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-10T03:32:05.902 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-10T03:32:05.902 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-10T03:32:06.053 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-10T03:32:06.057 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-10T03:32:06.106 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-10T03:32:06.140 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-10T03:32:06.154 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:06.188 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-10T03:32:06.210 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 13/13 2024-09-10T03:32:06.612 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:06.874 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:06.874 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:06.874 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-10T03:32:06.874 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:32:06.874 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:06.874 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:32:06.955 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 13/13 2024-09-10T03:32:06.956 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 1/13 2024-09-10T03:32:06.956 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-10T03:32:06.956 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-10T03:32:06.956 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-10T03:32:06.956 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-10T03:32:06.956 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-10T03:32:06.956 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-10T03:32:06.956 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-10T03:32:06.956 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-10T03:32:06.956 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-10T03:32:06.956 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-10T03:32:06.957 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-10T03:32:07.141 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs2 2024-09-10T03:32:07.281 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-10T03:32:07.282 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:07.282 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-10T03:32:07.282 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-10T03:32:07.282 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T03:32:07.282 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T03:32:07.282 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T03:32:07.282 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T03:32:07.282 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T03:32:07.282 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T03:32:07.282 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T03:32:07.282 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T03:32:07.282 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T03:32:07.282 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T03:32:07.283 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T03:32:07.283 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T03:32:07.283 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:07.283 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:32:07.538 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-cephadm 2024-09-10T03:32:07.721 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:38 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:32:07.813 INFO:teuthology.orchestra.run.smithi112.stdout:Package libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:32:07.857 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:32:07.858 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-10T03:32:07.858 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:32:07.987 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs-devel 2024-09-10T03:32:08.100 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:23 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:32:08.234 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:32:08.235 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:08.235 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-10T03:32:08.235 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:08.235 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-10T03:32:08.235 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 146 k 2024-09-10T03:32:08.235 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-10T03:32:08.236 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T03:32:08.236 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T03:32:08.236 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T03:32:08.236 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T03:32:08.236 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T03:32:08.236 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:08.236 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-10T03:32:08.236 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:08.236 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2024-09-10T03:32:08.236 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:08.237 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 7.0 M 2024-09-10T03:32:08.237 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 33 M 2024-09-10T03:32:08.237 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-10T03:32:08.545 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:39 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:32:08.680 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:32:08.680 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-09-10T03:32:08.681 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:08.681 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-10T03:32:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 32 k 2024-09-10T03:32:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:08.681 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-10T03:32:08.681 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:08.681 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-09-10T03:32:08.681 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:08.682 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 32 k 2024-09-10T03:32:08.682 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 142 k 2024-09-10T03:32:08.682 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-10T03:32:08.827 INFO:teuthology.orchestra.run.smithi112.stdout:libcephfs-devel-19.1.1-298.gb53b6d3e.el9.x86_64 218 kB/s | 32 kB 00:00 2024-09-10T03:32:08.827 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:32:08.827 INFO:teuthology.orchestra.run.smithi112.stdout:Total 216 kB/s | 32 kB 00:00 2024-09-10T03:32:08.828 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-10T03:32:08.831 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-10T03:32:08.831 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-10T03:32:08.847 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-10T03:32:08.848 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-10T03:32:08.883 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-10T03:32:08.933 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-10T03:32:08.941 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-10T03:32:08.967 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): ceph-mgr-cephadm-19.1.1-298.gb53b6d3e.el 647 kB/s | 146 kB 00:00 2024-09-10T03:32:08.992 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-10T03:32:09.032 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:09.034 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-10T03:32:09.411 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:09.652 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:09.653 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:09.653 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-10T03:32:09.653 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:32:09.653 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:09.653 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:32:09.826 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados2 2024-09-10T03:32:10.226 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.0 MB/s | 6.0 MB 00:01 2024-09-10T03:32:10.228 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:32:10.229 INFO:teuthology.orchestra.run.smithi169.stdout:Total 3.5 MB/s | 7.0 MB 00:01 2024-09-10T03:32:10.302 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-10T03:32:10.314 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-10T03:32:10.314 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-10T03:32:10.382 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:41 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:32:10.413 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-10T03:32:10.413 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-10T03:32:10.472 INFO:teuthology.orchestra.run.smithi112.stdout:Package librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:32:10.515 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:32:10.516 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-10T03:32:10.516 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:32:10.599 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librbd1 2024-09-10T03:32:10.729 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-10T03:32:10.909 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-10T03:32:10.961 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-10T03:32:11.158 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:42 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:32:11.249 INFO:teuthology.orchestra.run.smithi112.stdout:Package librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:32:11.292 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:32:11.293 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-10T03:32:11.293 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:32:11.373 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rados 2024-09-10T03:32:11.488 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-10T03:32:11.580 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-10T03:32:11.644 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-10T03:32:11.673 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 6/6 2024-09-10T03:32:11.938 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:42 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:32:12.028 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:32:12.071 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:32:12.072 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-10T03:32:12.072 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:32:12.152 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rgw 2024-09-10T03:32:12.476 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 6/6 2024-09-10T03:32:12.476 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/6 2024-09-10T03:32:12.476 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-10T03:32:12.476 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-10T03:32:12.476 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-10T03:32:12.476 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-10T03:32:12.714 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:43 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:32:12.737 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-10T03:32:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:12.738 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-10T03:32:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-10T03:32:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T03:32:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T03:32:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T03:32:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T03:32:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T03:32:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:12.738 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:32:12.804 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:32:12.848 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:32:12.849 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-10T03:32:12.850 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:32:12.934 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-cephfs 2024-09-10T03:32:12.964 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-fuse 2024-09-10T03:32:13.495 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:44 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:32:13.535 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:28 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:32:13.588 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:32:13.631 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:32:13.632 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-10T03:32:13.632 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:32:13.670 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:32:13.670 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:13.670 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-10T03:32:13.670 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:13.670 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-10T03:32:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 897 k 2024-09-10T03:32:13.671 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-10T03:32:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T03:32:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:13.671 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-10T03:32:13.671 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:13.671 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2024-09-10T03:32:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:13.672 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 977 k 2024-09-10T03:32:13.672 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 2.9 M 2024-09-10T03:32:13.672 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-10T03:32:13.710 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rbd 2024-09-10T03:32:13.969 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): ceph-fuse-19.1.1-298.gb53b6d3e.el9.x86_6 3.2 MB/s | 897 kB 00:00 2024-09-10T03:32:14.119 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 187 kB/s | 80 kB 00:00 2024-09-10T03:32:14.120 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:32:14.120 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.1 MB/s | 977 kB 00:00 2024-09-10T03:32:14.132 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-10T03:32:14.154 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-10T03:32:14.154 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-10T03:32:14.236 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-10T03:32:14.237 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-10T03:32:14.274 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:45 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:32:14.364 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:32:14.407 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:32:14.408 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-10T03:32:14.408 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:32:14.423 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-10T03:32:14.486 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-fuse 2024-09-10T03:32:14.564 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-10T03:32:14.644 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/2 2024-09-10T03:32:15.049 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:46 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:32:15.183 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:32:15.183 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:15.183 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-09-10T03:32:15.183 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:15.184 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-10T03:32:15.184 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 85 k 2024-09-10T03:32:15.184 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:15.184 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-10T03:32:15.184 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:15.184 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-09-10T03:32:15.184 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:15.185 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 85 k 2024-09-10T03:32:15.185 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 231 k 2024-09-10T03:32:15.185 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-10T03:32:15.336 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/2 2024-09-10T03:32:15.336 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/2 2024-09-10T03:32:15.388 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-fuse-19.1.1-298.gb53b6d3e.el9.x86_64.rpm 419 kB/s | 85 kB 00:00 2024-09-10T03:32:15.388 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:32:15.389 INFO:teuthology.orchestra.run.smithi112.stdout:Total 416 kB/s | 85 kB 00:00 2024-09-10T03:32:15.389 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-10T03:32:15.395 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-10T03:32:15.396 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-10T03:32:15.461 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-10T03:32:15.461 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-10T03:32:15.606 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-10T03:32:15.631 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-10T03:32:15.631 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:15.631 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-10T03:32:15.631 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-10T03:32:15.631 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:15.631 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:32:15.710 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:15.952 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados-devel 2024-09-10T03:32:16.100 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:16.374 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:16.374 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:16.374 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-10T03:32:16.374 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:32:16.374 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:16.374 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:32:16.531 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:31 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:32:16.639 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-mirror 2024-09-10T03:32:16.671 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:32:16.672 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:16.672 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-09-10T03:32:16.672 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:16.672 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-10T03:32:16.672 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 127 k 2024-09-10T03:32:16.672 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:16.672 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-10T03:32:16.672 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:16.673 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-09-10T03:32:16.673 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:16.673 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 127 k 2024-09-10T03:32:16.673 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 456 k 2024-09-10T03:32:16.673 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-10T03:32:16.891 INFO:teuthology.orchestra.run.smithi169.stdout:librados-devel-19.1.1-298.gb53b6d3e.el9.x86_64. 583 kB/s | 127 kB 00:00 2024-09-10T03:32:16.891 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:32:16.892 INFO:teuthology.orchestra.run.smithi169.stdout:Total 578 kB/s | 127 kB 00:00 2024-09-10T03:32:16.892 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-10T03:32:16.897 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-10T03:32:16.898 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-10T03:32:16.962 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-10T03:32:16.963 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-10T03:32:17.114 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-10T03:32:17.206 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:17.219 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:48 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:32:17.355 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:32:17.356 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:17.356 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-09-10T03:32:17.356 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:17.356 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-10T03:32:17.357 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.1 M 2024-09-10T03:32:17.357 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:17.357 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-10T03:32:17.357 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:17.357 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-09-10T03:32:17.357 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:17.357 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 3.1 M 2024-09-10T03:32:17.357 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 13 M 2024-09-10T03:32:17.357 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-10T03:32:17.634 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:17.724 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-mirror-19.1.1-298.gb53b6d3e.el9.x86_64.rpm 8.4 MB/s | 3.1 MB 00:00 2024-09-10T03:32:17.725 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:32:17.725 INFO:teuthology.orchestra.run.smithi112.stdout:Total 8.4 MB/s | 3.1 MB 00:00 2024-09-10T03:32:17.725 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-10T03:32:17.734 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-10T03:32:17.734 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-10T03:32:17.815 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-10T03:32:17.815 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-10T03:32:17.854 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:17.854 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:17.854 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-10T03:32:17.854 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:32:17.854 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:17.854 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:32:18.144 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs2 2024-09-10T03:32:18.245 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-10T03:32:18.316 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:18.340 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:18.341 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T03:32:18.341 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T03:32:18.341 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T03:32:18.341 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T03:32:18.341 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:18.728 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:33 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:32:18.822 INFO:teuthology.orchestra.run.smithi169.stdout:Package libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:32:18.867 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:32:18.868 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-10T03:32:18.868 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:32:18.929 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs-devel 2024-09-10T03:32:19.435 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:19.435 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:19.435 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-10T03:32:19.435 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:32:19.435 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:19.435 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:32:19.498 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:34 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:32:19.631 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:32:19.632 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:19.632 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-09-10T03:32:19.632 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:19.632 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-10T03:32:19.633 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 32 k 2024-09-10T03:32:19.633 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:19.633 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-10T03:32:19.633 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:19.633 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-09-10T03:32:19.633 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:19.633 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 32 k 2024-09-10T03:32:19.633 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 142 k 2024-09-10T03:32:19.633 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-10T03:32:19.705 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-nbd 2024-09-10T03:32:19.786 INFO:teuthology.orchestra.run.smithi169.stdout:libcephfs-devel-19.1.1-298.gb53b6d3e.el9.x86_64 207 kB/s | 32 kB 00:00 2024-09-10T03:32:19.787 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:32:19.787 INFO:teuthology.orchestra.run.smithi169.stdout:Total 205 kB/s | 32 kB 00:00 2024-09-10T03:32:19.787 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-10T03:32:19.790 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-10T03:32:19.791 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-10T03:32:19.811 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-10T03:32:19.811 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-10T03:32:19.904 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-10T03:32:20.001 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:20.289 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:51 ago on Tue 10 Sep 2024 03:30:29 AM UTC. 2024-09-10T03:32:20.361 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:20.425 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-10T03:32:20.426 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:20.426 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-09-10T03:32:20.426 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:20.426 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-10T03:32:20.426 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 171 k 2024-09-10T03:32:20.426 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:20.426 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-10T03:32:20.426 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-10T03:32:20.426 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-09-10T03:32:20.426 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:20.426 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 171 k 2024-09-10T03:32:20.426 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 494 k 2024-09-10T03:32:20.426 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-10T03:32:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:20.594 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-10T03:32:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:32:20.594 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:20.594 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:32:20.636 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-nbd-19.1.1-298.gb53b6d3e.el9.x86_64.rpm 818 kB/s | 171 kB 00:00 2024-09-10T03:32:20.636 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:32:20.636 INFO:teuthology.orchestra.run.smithi112.stdout:Total 812 kB/s | 171 kB 00:00 2024-09-10T03:32:20.637 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-10T03:32:20.644 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-10T03:32:20.644 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-10T03:32:20.708 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-10T03:32:20.709 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-10T03:32:20.776 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados2 2024-09-10T03:32:20.872 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-10T03:32:20.974 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:21.346 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:36 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:32:21.437 INFO:teuthology.orchestra.run.smithi169.stdout:Package librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:32:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:21.481 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:32:21.482 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-10T03:32:21.482 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:32:21.548 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librbd1 2024-09-10T03:32:21.738 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:21.738 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:21.738 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-10T03:32:21.739 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:32:21.739 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:32:21.739 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-10T03:32:22.027 DEBUG:teuthology.parallel:result is None 2024-09-10T03:32:22.113 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:37 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:32:22.203 INFO:teuthology.orchestra.run.smithi169.stdout:Package librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:32:22.248 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:32:22.249 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-10T03:32:22.249 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:32:22.319 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rados 2024-09-10T03:32:22.891 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:37 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:32:22.982 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:32:23.026 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:32:23.026 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-10T03:32:23.027 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:32:23.094 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rgw 2024-09-10T03:32:23.661 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:38 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:32:23.751 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:32:23.796 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:32:23.797 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-10T03:32:23.797 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:32:23.862 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-cephfs 2024-09-10T03:32:24.429 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:39 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:32:24.520 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:32:24.563 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:32:24.564 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-10T03:32:24.565 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:32:24.640 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rbd 2024-09-10T03:32:25.204 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:40 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:32:25.294 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-10T03:32:25.337 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:32:25.338 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-10T03:32:25.338 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:32:25.398 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-fuse 2024-09-10T03:32:25.964 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:40 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:32:26.097 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:32:26.098 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:26.098 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-09-10T03:32:26.098 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:26.098 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-10T03:32:26.098 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 85 k 2024-09-10T03:32:26.098 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:26.098 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-10T03:32:26.098 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:26.099 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-09-10T03:32:26.099 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:26.099 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 85 k 2024-09-10T03:32:26.099 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 231 k 2024-09-10T03:32:26.099 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-10T03:32:26.289 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-fuse-19.1.1-298.gb53b6d3e.el9.x86_64.rpm 448 kB/s | 85 kB 00:00 2024-09-10T03:32:26.290 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:32:26.290 INFO:teuthology.orchestra.run.smithi169.stdout:Total 444 kB/s | 85 kB 00:00 2024-09-10T03:32:26.290 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-10T03:32:26.297 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-10T03:32:26.297 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-10T03:32:26.361 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-10T03:32:26.361 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-10T03:32:26.508 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-10T03:32:26.621 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:26.990 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:27.204 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:27.204 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:27.205 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-10T03:32:27.205 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:32:27.205 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:27.205 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:32:27.426 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-mirror 2024-09-10T03:32:28.005 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:43 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:32:28.143 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:32:28.144 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:28.144 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-09-10T03:32:28.144 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:28.145 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-10T03:32:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.1 M 2024-09-10T03:32:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:28.145 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-10T03:32:28.145 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:28.145 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-09-10T03:32:28.145 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:28.145 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 3.1 M 2024-09-10T03:32:28.145 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 13 M 2024-09-10T03:32:28.145 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-10T03:32:28.467 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-mirror-19.1.1-298.gb53b6d3e.el9.x86_64.rpm 9.6 MB/s | 3.1 MB 00:00 2024-09-10T03:32:28.468 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:32:28.468 INFO:teuthology.orchestra.run.smithi169.stdout:Total 9.6 MB/s | 3.1 MB 00:00 2024-09-10T03:32:28.469 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-10T03:32:28.477 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-10T03:32:28.477 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-10T03:32:28.558 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-10T03:32:28.559 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-10T03:32:28.992 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-10T03:32:29.052 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:29.077 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:29.077 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T03:32:29.077 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T03:32:29.077 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T03:32:29.077 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T03:32:29.078 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:30.228 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:30.228 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:30.228 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-10T03:32:30.228 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:32:30.228 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:30.228 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:32:30.479 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-nbd 2024-09-10T03:32:31.054 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:46 ago on Tue 10 Sep 2024 03:30:45 AM UTC. 2024-09-10T03:32:31.197 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-10T03:32:31.197 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:31.197 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-09-10T03:32:31.198 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:31.198 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-10T03:32:31.198 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 171 k 2024-09-10T03:32:31.198 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:31.198 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-10T03:32:31.198 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-10T03:32:31.199 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-09-10T03:32:31.199 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:31.199 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 171 k 2024-09-10T03:32:31.199 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 494 k 2024-09-10T03:32:31.199 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-10T03:32:31.405 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-nbd-19.1.1-298.gb53b6d3e.el9.x86_64.rpm 823 kB/s | 171 kB 00:00 2024-09-10T03:32:31.406 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:32:31.406 INFO:teuthology.orchestra.run.smithi169.stdout:Total 816 kB/s | 171 kB 00:00 2024-09-10T03:32:31.406 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-10T03:32:31.414 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-10T03:32:31.415 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-10T03:32:31.482 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-10T03:32:31.482 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-10T03:32:31.656 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-10T03:32:31.764 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:32.180 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:32.451 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-10T03:32:32.451 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:32.452 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-10T03:32:32.452 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-10T03:32:32.452 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:32:32.452 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-10T03:32:32.676 DEBUG:teuthology.parallel:result is None 2024-09-10T03:32:32.677 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T03:32:32.677 INFO:teuthology.packaging:ref: None 2024-09-10T03:32:32.677 INFO:teuthology.packaging:tag: None 2024-09-10T03:32:32.677 INFO:teuthology.packaging:branch: squid 2024-09-10T03:32:32.677 INFO:teuthology.packaging:sha1: 3a5e3f16720f432aea965c40a8f6eb480095ec6c 2024-09-10T03:32:32.678 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-10T03:32:32.896 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T03:32:32.923 INFO:teuthology.orchestra.run.smithi112.stdout:19.1.1-298.gb53b6d3e.el9 2024-09-10T03:32:32.924 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-298.gb53b6d3e.el9 2024-09-10T03:32:32.924 INFO:teuthology.task.install:The correct ceph version 19.1.1-298.gb53b6d3e is installed. 2024-09-10T03:32:32.926 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T03:32:32.926 INFO:teuthology.packaging:ref: None 2024-09-10T03:32:32.926 INFO:teuthology.packaging:tag: None 2024-09-10T03:32:32.926 INFO:teuthology.packaging:branch: squid 2024-09-10T03:32:32.926 INFO:teuthology.packaging:sha1: 3a5e3f16720f432aea965c40a8f6eb480095ec6c 2024-09-10T03:32:32.926 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-10T03:32:33.128 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T03:32:33.150 INFO:teuthology.orchestra.run.smithi169.stdout:19.1.1-298.gb53b6d3e.el9 2024-09-10T03:32:33.150 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-298.gb53b6d3e.el9 2024-09-10T03:32:33.150 INFO:teuthology.task.install:The correct ceph version 19.1.1-298.gb53b6d3e is installed. 2024-09-10T03:32:33.152 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-10T03:32:33.153 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T03:32:33.153 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T03:32:33.184 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-10T03:32:33.185 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T03:32:33.216 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-10T03:32:33.217 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T03:32:33.217 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T03:32:33.252 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T03:32:33.320 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-10T03:32:33.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T03:32:33.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T03:32:33.425 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-10T03:32:33.425 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T03:32:33.426 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T03:32:33.454 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T03:32:33.522 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-10T03:32:33.522 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T03:32:33.547 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T03:32:33.612 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-10T03:32:33.612 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T03:32:33.613 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T03:32:33.640 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T03:32:33.707 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-10T03:32:33.707 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T03:32:33.731 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T03:32:33.796 INFO:teuthology.run_tasks:Running task print... 2024-09-10T03:32:33.806 INFO:teuthology.task.print:**** done install task... 2024-09-10T03:32:33.806 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-10T03:32:33.907 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'squid', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'global': {'mon pg warn min per osd': 0}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}}, 'image': 'quay.ceph.io/ceph-ci/ceph:squid', 'roleless': True, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'sha1': '3a5e3f16720f432aea965c40a8f6eb480095ec6c'} 2024-09-10T03:32:33.907 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:squid 2024-09-10T03:32:33.908 INFO:tasks.cephadm:Cluster fsid is 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:32:33.908 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-10T03:32:33.908 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-09-10T03:32:33.909 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi112': '172.21.15.112', 'mon.smithi169': '172.21.15.169'} 2024-09-10T03:32:33.909 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-10T03:32:33.909 DEBUG:teuthology.orchestra.run.smithi112:> sudo hostname $(hostname -s) 2024-09-10T03:32:33.936 DEBUG:teuthology.orchestra.run.smithi169:> sudo hostname $(hostname -s) 2024-09-10T03:32:33.961 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for squid 2024-09-10T03:32:33.961 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3a5e3f16720f432aea965c40a8f6eb480095ec6c 2024-09-10T03:32:34.109 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3a5e3f16720f432aea965c40a8f6eb480095ec6c', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/82931/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-4885-g3a5e3f16', 'node_name': '172.21.2.4+braggi04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-4885.g3a5e3f16'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-adk-testing-2024-09-09-1904/3a5e3f16720f432aea965c40a8f6eb480095ec6c/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-10 02:14:59.834535', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-adk-testing-2024-09-09-1904', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-adk-testing-2024-09-09-1904/3a5e3f16720f432aea965c40a8f6eb480095ec6c/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-09-10T03:32:34.348 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref squid, sha1 b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=squid 2024-09-10T03:32:34.350 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/x86_64/flavors/default/cephadm 2024-09-10T03:32:34.350 INFO:tasks.cephadm:Downloading cephadm from url: https://2.chacra.ceph.com/binaries/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/x86_64/flavors/default/cephadm 2024-09-10T03:32:34.350 DEBUG:teuthology.orchestra.run.smithi112:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-10T03:32:34.748 INFO:teuthology.orchestra.run.smithi112.stdout:-rw-r--r--. 1 ubuntu ubuntu 784498 Sep 10 03:32 /home/ubuntu/cephtest/cephadm 2024-09-10T03:32:34.748 DEBUG:teuthology.orchestra.run.smithi169:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-10T03:32:35.111 INFO:teuthology.orchestra.run.smithi169.stdout:-rw-r--r--. 1 ubuntu ubuntu 784498 Sep 10 03:32 /home/ubuntu/cephtest/cephadm 2024-09-10T03:32:35.112 DEBUG:teuthology.orchestra.run.smithi112:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-10T03:32:35.132 DEBUG:teuthology.orchestra.run.smithi169:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-10T03:32:35.161 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:squid on all hosts... 2024-09-10T03:32:35.162 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid pull 2024-09-10T03:32:35.175 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid pull 2024-09-10T03:32:36.773 INFO:teuthology.orchestra.run.smithi112.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:squid... 2024-09-10T03:32:36.828 INFO:teuthology.orchestra.run.smithi169.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:squid... 2024-09-10T03:33:14.943 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-10T03:33:14.944 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_version": "ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable)", 2024-09-10T03:33:14.944 INFO:teuthology.orchestra.run.smithi112.stdout: "image_id": "54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594", 2024-09-10T03:33:14.944 INFO:teuthology.orchestra.run.smithi112.stdout: "repo_digests": [ 2024-09-10T03:33:14.944 INFO:teuthology.orchestra.run.smithi112.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66" 2024-09-10T03:33:14.944 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-10T03:33:14.944 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-10T03:33:22.181 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-10T03:33:22.181 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_version": "ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable)", 2024-09-10T03:33:22.181 INFO:teuthology.orchestra.run.smithi169.stdout: "image_id": "54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594", 2024-09-10T03:33:22.181 INFO:teuthology.orchestra.run.smithi169.stdout: "repo_digests": [ 2024-09-10T03:33:22.181 INFO:teuthology.orchestra.run.smithi169.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66" 2024-09-10T03:33:22.181 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-10T03:33:22.181 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-10T03:33:22.206 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph 2024-09-10T03:33:22.240 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph 2024-09-10T03:33:22.267 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /etc/ceph 2024-09-10T03:33:22.308 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 777 /etc/ceph 2024-09-10T03:33:22.332 INFO:tasks.cephadm:Writing seed config... 2024-09-10T03:33:22.334 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-09-10T03:33:22.334 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-09-10T03:33:22.334 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-09-10T03:33:22.334 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-09-10T03:33:22.334 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-09-10T03:33:22.334 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-09-10T03:33:22.334 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-09-10T03:33:22.334 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-09-10T03:33:22.334 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-09-10T03:33:22.334 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-10T03:33:22.334 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-10T03:33:22.335 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-09-10T03:33:22.335 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-09-10T03:33:22.335 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-09-10T03:33:22.335 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-09-10T03:33:22.335 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-09-10T03:33:22.335 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-09-10T03:33:22.335 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-09-10T03:33:22.336 INFO:tasks.cephadm: override: [mgr] client mount timeout = 30 2024-09-10T03:33:22.336 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-09-10T03:33:22.336 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-10T03:33:22.336 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-10T03:33:22.336 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-10T03:33:22.336 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-10T03:33:22.337 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-10T03:33:22.337 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-09-10T03:33:22.337 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T03:33:22.338 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-10T03:33:22.366 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 513b3bc6-6f25-11ef-bcea-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false client mount timeout = 30 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-10T03:33:22.367 DEBUG:teuthology.orchestra.run.smithi112:mon.smithi112> sudo journalctl -f -n 0 -u ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@mon.smithi112.service 2024-09-10T03:33:22.409 INFO:tasks.cephadm:Bootstrapping... 2024-09-10T03:33:22.409 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid -v bootstrap --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.112 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-10T03:33:22.615 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-10T03:33:22.615 INFO:teuthology.orchestra.run.smithi112.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', '-v', 'bootstrap', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.112', '--skip-admin-label'] 2024-09-10T03:33:22.615 INFO:teuthology.orchestra.run.smithi112.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-10T03:33:22.615 INFO:teuthology.orchestra.run.smithi112.stdout:Verifying podman|docker is present... 2024-09-10T03:33:22.649 INFO:teuthology.orchestra.run.smithi112.stdout:/bin/podman: stdout 5.2.2 2024-09-10T03:33:22.649 INFO:teuthology.orchestra.run.smithi112.stdout:Verifying lvm2 is present... 2024-09-10T03:33:22.649 INFO:teuthology.orchestra.run.smithi112.stdout:Verifying time synchronization is in place... 2024-09-10T03:33:22.657 INFO:teuthology.orchestra.run.smithi112.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-10T03:33:22.657 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-10T03:33:22.663 INFO:teuthology.orchestra.run.smithi112.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-10T03:33:22.663 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stdout inactive 2024-09-10T03:33:22.670 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stdout enabled 2024-09-10T03:33:22.676 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stdout active 2024-09-10T03:33:22.676 INFO:teuthology.orchestra.run.smithi112.stdout:Unit chronyd.service is enabled and running 2024-09-10T03:33:22.676 INFO:teuthology.orchestra.run.smithi112.stdout:Repeating the final host check... 2024-09-10T03:33:22.699 INFO:teuthology.orchestra.run.smithi112.stdout:/bin/podman: stdout 5.2.2 2024-09-10T03:33:22.699 INFO:teuthology.orchestra.run.smithi112.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-10T03:33:22.699 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl is present 2024-09-10T03:33:22.699 INFO:teuthology.orchestra.run.smithi112.stdout:lvcreate is present 2024-09-10T03:33:22.705 INFO:teuthology.orchestra.run.smithi112.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-10T03:33:22.706 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-10T03:33:22.712 INFO:teuthology.orchestra.run.smithi112.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-10T03:33:22.712 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stdout inactive 2024-09-10T03:33:22.718 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stdout enabled 2024-09-10T03:33:22.725 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stdout active 2024-09-10T03:33:22.725 INFO:teuthology.orchestra.run.smithi112.stdout:Unit chronyd.service is enabled and running 2024-09-10T03:33:22.725 INFO:teuthology.orchestra.run.smithi112.stdout:Host looks OK 2024-09-10T03:33:22.725 INFO:teuthology.orchestra.run.smithi112.stdout:Cluster fsid: 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:33:22.725 INFO:teuthology.orchestra.run.smithi112.stdout:Acquiring lock 139997881403232 on /run/cephadm/513b3bc6-6f25-11ef-bcea-c7b262605968.lock 2024-09-10T03:33:22.725 INFO:teuthology.orchestra.run.smithi112.stdout:Lock 139997881403232 acquired on /run/cephadm/513b3bc6-6f25-11ef-bcea-c7b262605968.lock 2024-09-10T03:33:22.725 INFO:teuthology.orchestra.run.smithi112.stdout:Verifying IP 172.21.15.112 port 3300 ... 2024-09-10T03:33:22.726 INFO:teuthology.orchestra.run.smithi112.stdout:Verifying IP 172.21.15.112 port 6789 ... 2024-09-10T03:33:22.726 INFO:teuthology.orchestra.run.smithi112.stdout:Base mon IP(s) is [172.21.15.112:3300, 172.21.15.112:6789], mon addrv is [v2:172.21.15.112:3300,v1:172.21.15.112:6789] 2024-09-10T03:33:22.729 INFO:teuthology.orchestra.run.smithi112.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.112 metric 100 2024-09-10T03:33:22.729 INFO:teuthology.orchestra.run.smithi112.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.112 metric 100 2024-09-10T03:33:22.732 INFO:teuthology.orchestra.run.smithi112.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-10T03:33:22.732 INFO:teuthology.orchestra.run.smithi112.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-10T03:33:22.732 INFO:teuthology.orchestra.run.smithi112.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-10T03:33:22.734 INFO:teuthology.orchestra.run.smithi112.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-10T03:33:22.734 INFO:teuthology.orchestra.run.smithi112.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-10T03:33:22.734 INFO:teuthology.orchestra.run.smithi112.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-10T03:33:22.734 INFO:teuthology.orchestra.run.smithi112.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-10T03:33:22.734 INFO:teuthology.orchestra.run.smithi112.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cb69/64 scope link noprefixroute 2024-09-10T03:33:22.734 INFO:teuthology.orchestra.run.smithi112.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-10T03:33:22.735 INFO:teuthology.orchestra.run.smithi112.stdout:Mon IP `172.21.15.112` is in CIDR network `172.21.0.0/20` 2024-09-10T03:33:22.735 INFO:teuthology.orchestra.run.smithi112.stdout:Mon IP `172.21.15.112` is in CIDR network `172.21.0.0/20` 2024-09-10T03:33:22.735 INFO:teuthology.orchestra.run.smithi112.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-10T03:33:22.736 INFO:teuthology.orchestra.run.smithi112.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-10T03:33:22.736 INFO:teuthology.orchestra.run.smithi112.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:squid... 2024-09-10T03:33:23.228 INFO:teuthology.orchestra.run.smithi112.stdout:/bin/podman: stdout 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 2024-09-10T03:33:23.228 INFO:teuthology.orchestra.run.smithi112.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:squid... 2024-09-10T03:33:23.228 INFO:teuthology.orchestra.run.smithi112.stdout:/bin/podman: stderr Getting image source signatures 2024-09-10T03:33:23.228 INFO:teuthology.orchestra.run.smithi112.stdout:/bin/podman: stderr Copying blob sha256:60180723200353b4e13b39fd162aecaf1bc097c859c7c6537324229e97c78567 2024-09-10T03:33:23.228 INFO:teuthology.orchestra.run.smithi112.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-10T03:33:23.228 INFO:teuthology.orchestra.run.smithi112.stdout:/bin/podman: stderr Copying config sha256:54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 2024-09-10T03:33:23.228 INFO:teuthology.orchestra.run.smithi112.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-10T03:33:24.150 INFO:teuthology.orchestra.run.smithi112.stdout:ceph: stdout ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable) 2024-09-10T03:33:24.151 INFO:teuthology.orchestra.run.smithi112.stdout:Ceph version: ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable) 2024-09-10T03:33:24.151 INFO:teuthology.orchestra.run.smithi112.stdout:Extracting ceph user uid/gid from container image... 2024-09-10T03:33:25.106 INFO:teuthology.orchestra.run.smithi112.stdout:stat: stdout 167 167 2024-09-10T03:33:25.106 INFO:teuthology.orchestra.run.smithi112.stdout:Creating initial keys... 2024-09-10T03:33:26.103 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph-authtool: stdout AQAFvt9m3b+0GRAAo9ZAV8Nb8zwcyXgoTotjxg== 2024-09-10T03:33:26.985 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph-authtool: stdout AQAGvt9m5VoYHBAA0MEmbbLmct/UlpmJ95gR/w== 2024-09-10T03:33:27.891 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph-authtool: stdout AQAHvt9mlHYAEhAASq+Ed6+fdxIVX8cKzpDMRA== 2024-09-10T03:33:27.892 INFO:teuthology.orchestra.run.smithi112.stdout:Creating initial monmap... 2024-09-10T03:33:28.839 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-10T03:33:28.840 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-10T03:33:28.840 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:33:28.840 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-10T03:33:28.840 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool for smithi112 [v2:172.21.15.112:3300,v1:172.21.15.112:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-10T03:33:28.840 INFO:teuthology.orchestra.run.smithi112.stdout:setting min_mon_release = quincy 2024-09-10T03:33:28.840 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/monmaptool: set fsid to 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:33:28.840 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-10T03:33:28.840 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:33:28.840 INFO:teuthology.orchestra.run.smithi112.stdout:Creating mon... 2024-09-10T03:33:29.695 INFO:teuthology.orchestra.run.smithi112.stdout:create mon.smithi112 on 2024-09-10T03:33:29.894 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-10T03:33:30.031 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-10T03:33:30.212 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-513b3bc6-6f25-11ef-bcea-c7b262605968.target → /etc/systemd/system/ceph-513b3bc6-6f25-11ef-bcea-c7b262605968.target. 2024-09-10T03:33:30.212 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-513b3bc6-6f25-11ef-bcea-c7b262605968.target → /etc/systemd/system/ceph-513b3bc6-6f25-11ef-bcea-c7b262605968.target. 2024-09-10T03:33:30.470 INFO:teuthology.orchestra.run.smithi112.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@mon.smithi112 2024-09-10T03:33:30.470 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stderr Failed to reset failed state of unit ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@mon.smithi112.service: Unit ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@mon.smithi112.service not loaded. 2024-09-10T03:33:30.634 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-513b3bc6-6f25-11ef-bcea-c7b262605968.target.wants/ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@mon.smithi112.service → /etc/systemd/system/ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@.service. 2024-09-10T03:33:31.109 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:31 smithi112 podman[27349]: 2024-09-10 03:33:31.067921473 +0000 UTC m=+0.323305445 container start fdf1303095406556fc5a635c3fe4a3957aef17ff0ae819ebaea41556233bc3d8 (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-10T03:33:31.109 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:31 smithi112 ceph-mon[27369]: mkfs 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:33:31.151 INFO:teuthology.orchestra.run.smithi112.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-10T03:33:31.151 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stdout disabled 2024-09-10T03:33:31.159 INFO:teuthology.orchestra.run.smithi112.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-10T03:33:31.159 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stdout inactive 2024-09-10T03:33:31.159 INFO:teuthology.orchestra.run.smithi112.stdout:firewalld.service is not enabled 2024-09-10T03:33:31.159 INFO:teuthology.orchestra.run.smithi112.stdout:Not possible to enable service . firewalld.service is not available 2024-09-10T03:33:31.159 INFO:teuthology.orchestra.run.smithi112.stdout:Waiting for mon to start... 2024-09-10T03:33:31.160 INFO:teuthology.orchestra.run.smithi112.stdout:Waiting for mon... 2024-09-10T03:33:31.375 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:31 smithi112 ceph-mon[27369]: mon.smithi112 is new leader, mons smithi112 in quorum (ranks 0) 2024-09-10T03:33:31.375 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:31 smithi112 bash[27349]: fdf1303095406556fc5a635c3fe4a3957aef17ff0ae819ebaea41556233bc3d8 2024-09-10T03:33:31.375 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:31 smithi112 systemd[1]: Started Ceph mon.smithi112 for 513b3bc6-6f25-11ef-bcea-c7b262605968. 2024-09-10T03:33:32.205 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:32 smithi112 ceph-mon[27369]: mon.smithi112 is new leader, mons smithi112 in quorum (ranks 0) 2024-09-10T03:33:32.206 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:32 smithi112 ceph-mon[27369]: monmap epoch 1 2024-09-10T03:33:32.206 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:32 smithi112 ceph-mon[27369]: fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:33:32.206 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:32 smithi112 ceph-mon[27369]: last_changed 2024-09-10T03:33:28.238100+0000 2024-09-10T03:33:32.206 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:32 smithi112 ceph-mon[27369]: created 2024-09-10T03:33:28.238100+0000 2024-09-10T03:33:32.206 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:32 smithi112 ceph-mon[27369]: min_mon_release 19 (squid) 2024-09-10T03:33:32.206 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:32 smithi112 ceph-mon[27369]: election_strategy: 1 2024-09-10T03:33:32.206 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:32 smithi112 ceph-mon[27369]: 0: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.smithi112 2024-09-10T03:33:32.206 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:32 smithi112 ceph-mon[27369]: fsmap 2024-09-10T03:33:32.206 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:32 smithi112 ceph-mon[27369]: osdmap e1: 0 total, 0 up, 0 in 2024-09-10T03:33:32.206 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:32 smithi112 ceph-mon[27369]: mgrmap e1: no daemons active 2024-09-10T03:33:32.206 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:32 smithi112 ceph-mon[27369]: from='client.? 172.21.15.112:0/697173310' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-10T03:33:32.209 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout cluster: 2024-09-10T03:33:32.209 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout id: 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:33:32.209 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-10T03:33:32.210 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout 2024-09-10T03:33:32.210 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout services: 2024-09-10T03:33:32.210 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi112 (age 0.556815s) 2024-09-10T03:33:32.210 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-10T03:33:32.210 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-10T03:33:32.210 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout 2024-09-10T03:33:32.210 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout data: 2024-09-10T03:33:32.210 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-10T03:33:32.210 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-10T03:33:32.210 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-10T03:33:32.210 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout pgs: 2024-09-10T03:33:32.211 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout 2024-09-10T03:33:32.211 INFO:teuthology.orchestra.run.smithi112.stdout:mon is available 2024-09-10T03:33:32.211 INFO:teuthology.orchestra.run.smithi112.stdout:Assimilating anything we can from ceph.conf... 2024-09-10T03:33:33.181 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout 2024-09-10T03:33:33.183 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout [global] 2024-09-10T03:33:33.183 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout fsid = 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:33:33.183 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.112:3300,v1:172.21.15.112:6789] 2024-09-10T03:33:33.184 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-10T03:33:33.184 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-10T03:33:33.184 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-10T03:33:33.184 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-10T03:33:33.184 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout 2024-09-10T03:33:33.184 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-10T03:33:33.184 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-10T03:33:33.184 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout 2024-09-10T03:33:33.184 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout [osd] 2024-09-10T03:33:33.184 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-10T03:33:33.184 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-10T03:33:33.184 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-10T03:33:33.185 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-10T03:33:33.185 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-10T03:33:33.185 INFO:teuthology.orchestra.run.smithi112.stdout:Generating new minimal ceph.conf... 2024-09-10T03:33:33.279 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:33 smithi112 ceph-mon[27369]: from='client.? 172.21.15.112:0/47649711' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-10T03:33:33.279 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:33 smithi112 ceph-mon[27369]: from='client.? 172.21.15.112:0/47649711' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-10T03:33:34.193 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:34 smithi112 ceph-mon[27369]: from='client.? 172.21.15.112:0/2205307889' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:33:34.196 INFO:teuthology.orchestra.run.smithi112.stdout:Restarting the monitor... 2024-09-10T03:33:34.472 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:34 smithi112 systemd[1]: Stopping Ceph mon.smithi112 for 513b3bc6-6f25-11ef-bcea-c7b262605968... 2024-09-10T03:33:34.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27365]: 2024-09-10T03:33:34.471+0000 7f272cc06640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi112 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T03:33:34.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27365]: 2024-09-10T03:33:34.471+0000 7f272cc06640 -1 mon.smithi112@0(leader) e1 *** Got Signal Terminated *** 2024-09-10T03:33:34.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:34 smithi112 podman[27611]: 2024-09-10 03:33:34.635118103 +0000 UTC m=+0.272512827 container died fdf1303095406556fc5a635c3fe4a3957aef17ff0ae819ebaea41556233bc3d8 (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T03:33:36.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:36 smithi112 podman[27611]: 2024-09-10 03:33:36.007596342 +0000 UTC m=+1.644991067 container cleanup fdf1303095406556fc5a635c3fe4a3957aef17ff0ae819ebaea41556233bc3d8 (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-b53b6d3, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T03:33:36.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:36 smithi112 bash[27611]: ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112 2024-09-10T03:33:38.565 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:38 smithi112 podman[27623]: 2024-09-10 03:33:38.171316275 +0000 UTC m=+3.538380162 container remove fdf1303095406556fc5a635c3fe4a3957aef17ff0ae819ebaea41556233bc3d8 (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True) 2024-09-10T03:33:38.565 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:38 smithi112 systemd[1]: ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@mon.smithi112.service: Deactivated successfully. 2024-09-10T03:33:38.565 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:38 smithi112 systemd[1]: Stopped Ceph mon.smithi112 for 513b3bc6-6f25-11ef-bcea-c7b262605968. 2024-09-10T03:33:38.565 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:38 smithi112 systemd[1]: Starting Ceph mon.smithi112 for 513b3bc6-6f25-11ef-bcea-c7b262605968... 2024-09-10T03:33:38.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:38 smithi112 podman[27718]: 2024-09-10 03:33:38.466794027 +0000 UTC m=+0.015905585 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay.ceph.io/ceph-ci/ceph:squid 2024-09-10T03:33:40.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:39 smithi112 podman[27718]: 2024-09-10 03:33:39.690533847 +0000 UTC m=+1.239645400 container create 91597bdf09f78a2afbdfa498aa1f0ab74554b1e43940ff53c2f6a2020ac8687e (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112, RELEASE=squid-b53b6d3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-10T03:33:42.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 podman[27718]: 2024-09-10 03:33:41.586191125 +0000 UTC m=+3.135302681 container init 91597bdf09f78a2afbdfa498aa1f0ab74554b1e43940ff53c2f6a2020ac8687e (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-10T03:33:42.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 podman[27718]: 2024-09-10 03:33:41.590298872 +0000 UTC m=+3.139410422 container start 91597bdf09f78a2afbdfa498aa1f0ab74554b1e43940ff53c2f6a2020ac8687e (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112, RELEASE=squid-b53b6d3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-09-10T03:33:42.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: set uid:gid to 167:167 (ceph:ceph) 2024-09-10T03:33:42.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable), process ceph-mon, pid 2 2024-09-10T03:33:42.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: pidfile_write: ignore empty --pid-file 2024-09-10T03:33:42.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: load: jerasure load: lrc 2024-09-10T03:33:42.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: RocksDB version: 7.9.2 2024-09-10T03:33:42.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Git sha 0 2024-09-10T03:33:42.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Compile date 2024-09-09 16:23:28 2024-09-10T03:33:42.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: DB SUMMARY 2024-09-10T03:33:42.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: DB Session ID: WSGXJUOJ3TUWITMGGTZN 2024-09-10T03:33:42.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: CURRENT file: CURRENT 2024-09-10T03:33:42.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: IDENTITY file: IDENTITY 2024-09-10T03:33:42.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-10T03:33:42.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi112/store.db dir, Total Num: 1, files: 000008.sst 2024-09-10T03:33:42.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi112/store.db: 000009.log size: 95843 ; 2024-09-10T03:33:42.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.error_if_exists: 0 2024-09-10T03:33:42.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.create_if_missing: 0 2024-09-10T03:33:42.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.paranoid_checks: 1 2024-09-10T03:33:42.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-10T03:33:42.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-10T03:33:42.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-10T03:33:42.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.env: 0x55f266d2f940 2024-09-10T03:33:42.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.fs: PosixFileSystem 2024-09-10T03:33:42.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.info_log: 0x55f268fe58c0 2024-09-10T03:33:42.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_file_opening_threads: 16 2024-09-10T03:33:42.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.statistics: (nil) 2024-09-10T03:33:42.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.use_fsync: 0 2024-09-10T03:33:42.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_log_file_size: 0 2024-09-10T03:33:42.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-10T03:33:42.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-10T03:33:42.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.keep_log_file_num: 1000 2024-09-10T03:33:42.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.recycle_log_file_num: 0 2024-09-10T03:33:42.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.allow_fallocate: 1 2024-09-10T03:33:42.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.allow_mmap_reads: 0 2024-09-10T03:33:42.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.allow_mmap_writes: 0 2024-09-10T03:33:42.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.use_direct_reads: 0 2024-09-10T03:33:42.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-10T03:33:42.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.create_missing_column_families: 0 2024-09-10T03:33:42.084 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.db_log_dir: 2024-09-10T03:33:42.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.wal_dir: 2024-09-10T03:33:42.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-10T03:33:42.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-10T03:33:42.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-10T03:33:42.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-10T03:33:42.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-10T03:33:42.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-10T03:33:42.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.advise_random_on_open: 1 2024-09-10T03:33:42.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.db_write_buffer_size: 0 2024-09-10T03:33:42.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.write_buffer_manager: 0x55f268ff4dc0 2024-09-10T03:33:42.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-10T03:33:42.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-10T03:33:42.085 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-10T03:33:42.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.rate_limiter: (nil) 2024-09-10T03:33:42.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-10T03:33:42.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.wal_recovery_mode: 2 2024-09-10T03:33:42.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.enable_thread_tracking: 0 2024-09-10T03:33:42.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.enable_pipelined_write: 0 2024-09-10T03:33:42.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.unordered_write: 0 2024-09-10T03:33:42.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-10T03:33:42.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-10T03:33:42.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-10T03:33:42.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-10T03:33:42.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.row_cache: None 2024-09-10T03:33:42.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.wal_filter: None 2024-09-10T03:33:42.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-10T03:33:42.086 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.allow_ingest_behind: 0 2024-09-10T03:33:42.088 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.two_write_queues: 0 2024-09-10T03:33:42.088 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.manual_wal_flush: 0 2024-09-10T03:33:42.088 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.wal_compression: 0 2024-09-10T03:33:42.088 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.atomic_flush: 0 2024-09-10T03:33:42.088 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-10T03:33:42.088 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-10T03:33:42.088 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-10T03:33:42.088 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.log_readahead_size: 0 2024-09-10T03:33:42.088 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-10T03:33:42.088 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.best_efforts_recovery: 0 2024-09-10T03:33:42.088 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-10T03:33:42.089 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-10T03:33:42.089 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.allow_data_in_errors: 0 2024-09-10T03:33:42.089 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.db_host_id: __hostname__ 2024-09-10T03:33:42.089 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-10T03:33:42.089 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_background_jobs: 2 2024-09-10T03:33:42.089 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_background_compactions: -1 2024-09-10T03:33:42.089 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_subcompactions: 1 2024-09-10T03:33:42.089 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-10T03:33:42.089 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-10T03:33:42.089 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-10T03:33:42.089 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_total_wal_size: 0 2024-09-10T03:33:42.089 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-10T03:33:42.089 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-10T03:33:42.089 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-10T03:33:42.090 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-10T03:33:42.090 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_open_files: -1 2024-09-10T03:33:42.090 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.bytes_per_sync: 0 2024-09-10T03:33:42.090 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-10T03:33:42.090 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-10T03:33:42.090 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compaction_readahead_size: 0 2024-09-10T03:33:42.090 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_background_flushes: -1 2024-09-10T03:33:42.090 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Compression algorithms supported: 2024-09-10T03:33:42.090 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: kZSTD supported: 0 2024-09-10T03:33:42.090 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: kXpressCompression supported: 0 2024-09-10T03:33:42.090 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: kBZip2Compression supported: 0 2024-09-10T03:33:42.090 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-10T03:33:42.091 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: kLZ4Compression supported: 1 2024-09-10T03:33:42.091 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: kZlibCompression supported: 1 2024-09-10T03:33:42.091 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: kLZ4HCCompression supported: 1 2024-09-10T03:33:42.091 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: kSnappyCompression supported: 1 2024-09-10T03:33:42.091 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-10T03:33:42.091 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-10T03:33:42.091 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi112/store.db/MANIFEST-000010 2024-09-10T03:33:42.091 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-10T03:33:42.091 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-10T03:33:42.091 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.merge_operator: 2024-09-10T03:33:42.091 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compaction_filter: None 2024-09-10T03:33:42.091 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compaction_filter_factory: None 2024-09-10T03:33:42.091 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.sst_partitioner_factory: None 2024-09-10T03:33:42.092 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-10T03:33:42.092 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-10T03:33:42.092 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f268fe5520) 2024-09-10T03:33:42.092 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: cache_index_and_filter_blocks: 1 2024-09-10T03:33:42.092 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-10T03:33:42.092 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-10T03:33:42.092 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: pin_top_level_index_and_filter: 1 2024-09-10T03:33:42.092 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: index_type: 0 2024-09-10T03:33:42.092 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: data_block_index_type: 0 2024-09-10T03:33:42.092 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: index_shortening: 1 2024-09-10T03:33:42.092 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-10T03:33:42.092 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: checksum: 4 2024-09-10T03:33:42.092 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: no_block_cache: 0 2024-09-10T03:33:42.093 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: block_cache: 0x55f268fd91f0 2024-09-10T03:33:42.093 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: block_cache_name: BinnedLRUCache 2024-09-10T03:33:42.093 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: block_cache_options: 2024-09-10T03:33:42.093 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: capacity : 536870912 2024-09-10T03:33:42.093 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: num_shard_bits : 4 2024-09-10T03:33:42.093 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: strict_capacity_limit : 0 2024-09-10T03:33:42.093 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: high_pri_pool_ratio: 0.000 2024-09-10T03:33:42.093 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: block_cache_compressed: (nil) 2024-09-10T03:33:42.093 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: persistent_cache: (nil) 2024-09-10T03:33:42.093 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: block_size: 4096 2024-09-10T03:33:42.093 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: block_size_deviation: 10 2024-09-10T03:33:42.093 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: block_restart_interval: 16 2024-09-10T03:33:42.094 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: index_block_restart_interval: 1 2024-09-10T03:33:42.094 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: metadata_block_size: 4096 2024-09-10T03:33:42.094 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: partition_filters: 0 2024-09-10T03:33:42.094 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: use_delta_encoding: 1 2024-09-10T03:33:42.094 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: filter_policy: bloomfilter 2024-09-10T03:33:42.094 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: whole_key_filtering: 1 2024-09-10T03:33:42.094 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: verify_compression: 0 2024-09-10T03:33:42.094 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: read_amp_bytes_per_bit: 0 2024-09-10T03:33:42.094 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: format_version: 5 2024-09-10T03:33:42.094 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: enable_index_compression: 1 2024-09-10T03:33:42.094 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: block_align: 0 2024-09-10T03:33:42.094 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: max_auto_readahead_size: 262144 2024-09-10T03:33:42.094 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: prepopulate_block_cache: 0 2024-09-10T03:33:42.094 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: initial_auto_readahead_size: 8192 2024-09-10T03:33:42.095 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: num_file_reads_for_auto_readahead: 2 2024-09-10T03:33:42.095 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.write_buffer_size: 33554432 2024-09-10T03:33:42.095 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_write_buffer_number: 2 2024-09-10T03:33:42.095 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compression: NoCompression 2024-09-10T03:33:42.095 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.bottommost_compression: Disabled 2024-09-10T03:33:42.095 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.prefix_extractor: nullptr 2024-09-10T03:33:42.095 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-10T03:33:42.095 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.num_levels: 7 2024-09-10T03:33:42.095 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-10T03:33:42.095 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-10T03:33:42.095 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-10T03:33:42.095 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-10T03:33:42.095 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-10T03:33:42.095 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-10T03:33:42.096 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-10T03:33:42.096 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-10T03:33:42.096 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-10T03:33:42.096 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-10T03:33:42.096 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-10T03:33:42.096 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-10T03:33:42.096 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-10T03:33:42.096 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compression_opts.level: 32767 2024-09-10T03:33:42.096 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compression_opts.strategy: 0 2024-09-10T03:33:42.096 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-10T03:33:42.096 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-10T03:33:42.096 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-10T03:33:42.096 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-10T03:33:42.097 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compression_opts.enabled: false 2024-09-10T03:33:42.097 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-10T03:33:42.097 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-10T03:33:42.097 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-10T03:33:42.097 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-10T03:33:42.097 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.target_file_size_base: 67108864 2024-09-10T03:33:42.097 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-10T03:33:42.097 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-10T03:33:42.097 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-10T03:33:42.097 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-10T03:33:42.097 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-10T03:33:42.097 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-10T03:33:42.097 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-10T03:33:42.097 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-10T03:33:42.098 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-10T03:33:42.098 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-10T03:33:42.098 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-10T03:33:42.098 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-10T03:33:42.098 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-10T03:33:42.098 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-10T03:33:42.098 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.arena_block_size: 1048576 2024-09-10T03:33:42.098 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-10T03:33:42.098 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-10T03:33:42.098 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.disable_auto_compactions: 0 2024-09-10T03:33:42.098 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-10T03:33:42.098 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-10T03:33:42.098 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-10T03:33:42.098 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-10T03:33:42.098 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-10T03:33:42.099 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-10T03:33:42.099 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-10T03:33:42.099 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-10T03:33:42.099 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-10T03:33:42.099 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-10T03:33:42.099 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-10T03:33:42.099 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.inplace_update_support: 0 2024-09-10T03:33:42.099 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-10T03:33:42.099 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-10T03:33:42.099 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-10T03:33:42.099 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-10T03:33:42.099 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.bloom_locality: 0 2024-09-10T03:33:42.099 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.max_successive_merges: 0 2024-09-10T03:33:42.099 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-10T03:33:42.100 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.paranoid_file_checks: 0 2024-09-10T03:33:42.100 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.force_consistency_checks: 1 2024-09-10T03:33:42.100 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.report_bg_io_stats: 0 2024-09-10T03:33:42.100 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.ttl: 2592000 2024-09-10T03:33:42.100 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-10T03:33:42.100 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-10T03:33:42.100 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-10T03:33:42.100 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.enable_blob_files: false 2024-09-10T03:33:42.100 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.min_blob_size: 0 2024-09-10T03:33:42.100 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.blob_file_size: 268435456 2024-09-10T03:33:42.100 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-10T03:33:42.100 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-10T03:33:42.100 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-10T03:33:42.101 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-10T03:33:42.101 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-10T03:33:42.101 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.blob_file_starting_level: 0 2024-09-10T03:33:42.101 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-10T03:33:42.101 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi112/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-10T03:33:42.101 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-10T03:33:42.101 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: af801dcd-6ea4-4877-b222-df3ca9a0c756 2024-09-10T03:33:42.101 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725939221618544, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-10T03:33:42.101 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-10T03:33:42.101 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725939221619558, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 91844, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 288, "table_properties": {"data_size": 89885, "index_size": 279, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11912, "raw_average_key_size": 48, "raw_value_size": 83253, "raw_average_value_size": 337, "num_data_blocks": 12, "num_entries": 247, "num_filter_entries": 247, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1725939221, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "af801dcd-6ea4-4877-b222-df3ca9a0c756", "db_session_id": "WSGXJUOJ3TUWITMGGTZN", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-10T03:33:42.101 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725939221619659, "job": 1, "event": "recovery_finished"} 2024-09-10T03:33:42.101 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-10T03:33:42.101 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi112/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-10T03:33:42.101 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f269006e00 2024-09-10T03:33:42.102 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: DB pointer 0x55f269102000 2024-09-10T03:33:42.102 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-10T03:33:42.102 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-10T03:33:42.102 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: ** DB Stats ** 2024-09-10T03:33:42.102 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T03:33:42.102 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-10T03:33:42.102 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-10T03:33:42.102 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T03:33:42.102 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-10T03:33:42.102 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-10T03:33:42.102 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T03:33:42.102 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: 2024-09-10T03:33:42.102 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: ** Compaction Stats [default] ** 2024-09-10T03:33:42.102 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-10T03:33:42.103 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-10T03:33:42.103 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: L0 2/0 91.59 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-10T03:33:42.103 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Sum 2/0 91.59 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-10T03:33:42.103 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-10T03:33:42.103 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: 2024-09-10T03:33:42.103 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: ** Compaction Stats [default] ** 2024-09-10T03:33:42.103 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-10T03:33:42.103 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T03:33:42.103 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 128.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-10T03:33:42.103 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: 2024-09-10T03:33:42.103 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-10T03:33:42.103 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: 2024-09-10T03:33:42.103 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T03:33:42.103 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T03:33:42.103 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T03:33:42.103 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T03:33:42.104 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T03:33:42.104 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-10T03:33:42.104 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Cumulative compaction: 0.00 GB write, 26.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T03:33:42.104 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Interval compaction: 0.00 GB write, 26.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T03:33:42.104 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T03:33:42.104 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Block cache BinnedLRUCache@0x55f268fd91f0#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-10T03:33:42.104 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-10T03:33:42.104 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: 2024-09-10T03:33:42.104 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-10T03:33:42.104 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: starting mon.smithi112 rank 0 at public addrs [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] at bind addrs [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi112 fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:33:42.104 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: mon.smithi112@-1(???) e1 preinit fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:33:42.104 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: mon.smithi112@-1(???).mds e1 new map 2024-09-10T03:33:42.104 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: mon.smithi112@-1(???).mds e1 print_map 2024-09-10T03:33:42.104 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: e1 2024-09-10T03:33:42.104 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: btime 2024-09-10T03:33:31:106056+0000 2024-09-10T03:33:42.105 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-10T03:33:42.105 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-10T03:33:42.105 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: legacy client fscid: -1 2024-09-10T03:33:42.105 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: 2024-09-10T03:33:42.105 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: No filesystems configured 2024-09-10T03:33:42.105 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: mon.smithi112@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-10T03:33:42.105 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: mon.smithi112@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T03:33:42.105 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: mon.smithi112@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T03:33:42.105 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: mon.smithi112@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T03:33:42.105 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: mon.smithi112@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-10T03:33:42.105 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: mon.smithi112 is new leader, mons smithi112 in quorum (ranks 0) 2024-09-10T03:33:42.105 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: monmap epoch 1 2024-09-10T03:33:42.105 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:33:42.105 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: last_changed 2024-09-10T03:33:28.238100+0000 2024-09-10T03:33:42.105 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: created 2024-09-10T03:33:28.238100+0000 2024-09-10T03:33:42.106 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: min_mon_release 19 (squid) 2024-09-10T03:33:42.106 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: election_strategy: 1 2024-09-10T03:33:42.106 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: 0: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.smithi112 2024-09-10T03:33:42.106 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: fsmap 2024-09-10T03:33:42.106 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: osdmap e1: 0 total, 0 up, 0 in 2024-09-10T03:33:42.106 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:41 smithi112 ceph-mon[27737]: mgrmap e1: no daemons active 2024-09-10T03:33:42.614 INFO:teuthology.orchestra.run.smithi112.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-10T03:33:43.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:42 smithi112 bash[27718]: 91597bdf09f78a2afbdfa498aa1f0ab74554b1e43940ff53c2f6a2020ac8687e 2024-09-10T03:33:43.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:42 smithi112 systemd[1]: Started Ceph mon.smithi112 for 513b3bc6-6f25-11ef-bcea-c7b262605968. 2024-09-10T03:33:46.149 INFO:teuthology.orchestra.run.smithi112.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-10T03:33:46.179 INFO:teuthology.orchestra.run.smithi112.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-10T03:33:46.179 INFO:teuthology.orchestra.run.smithi112.stdout:Creating mgr... 2024-09-10T03:33:46.179 INFO:teuthology.orchestra.run.smithi112.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-10T03:33:46.180 INFO:teuthology.orchestra.run.smithi112.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-10T03:33:46.180 INFO:teuthology.orchestra.run.smithi112.stdout:Verifying port 0.0.0.0:8443 ... 2024-09-10T03:33:46.431 INFO:teuthology.orchestra.run.smithi112.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@mgr.smithi112.pqrlrb 2024-09-10T03:33:46.431 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stderr Failed to reset failed state of unit ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@mgr.smithi112.pqrlrb.service: Unit ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@mgr.smithi112.pqrlrb.service not loaded. 2024-09-10T03:33:46.591 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-513b3bc6-6f25-11ef-bcea-c7b262605968.target.wants/ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@mgr.smithi112.pqrlrb.service → /etc/systemd/system/ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@.service. 2024-09-10T03:33:46.936 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:46 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3147624719' entity='client.admin' 2024-09-10T03:33:47.131 INFO:teuthology.orchestra.run.smithi112.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-10T03:33:47.131 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stdout disabled 2024-09-10T03:33:47.138 INFO:teuthology.orchestra.run.smithi112.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-10T03:33:47.138 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stdout inactive 2024-09-10T03:33:47.138 INFO:teuthology.orchestra.run.smithi112.stdout:firewalld.service is not enabled 2024-09-10T03:33:47.138 INFO:teuthology.orchestra.run.smithi112.stdout:Not possible to enable service . firewalld.service is not available 2024-09-10T03:33:47.147 INFO:teuthology.orchestra.run.smithi112.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-10T03:33:47.147 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stdout disabled 2024-09-10T03:33:47.154 INFO:teuthology.orchestra.run.smithi112.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-10T03:33:47.154 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stdout inactive 2024-09-10T03:33:47.154 INFO:teuthology.orchestra.run.smithi112.stdout:firewalld.service is not enabled 2024-09-10T03:33:47.154 INFO:teuthology.orchestra.run.smithi112.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-09-10T03:33:47.154 INFO:teuthology.orchestra.run.smithi112.stdout:Waiting for mgr to start... 2024-09-10T03:33:47.155 INFO:teuthology.orchestra.run.smithi112.stdout:Waiting for mgr... 2024-09-10T03:33:47.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:47 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/1292387810' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-10T03:33:48.321 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout 2024-09-10T03:33:48.321 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout { 2024-09-10T03:33:48.321 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "fsid": "513b3bc6-6f25-11ef-bcea-c7b262605968", 2024-09-10T03:33:48.321 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "health": { 2024-09-10T03:33:48.321 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-10T03:33:48.321 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-10T03:33:48.321 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-10T03:33:48.321 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:48.321 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-10T03:33:48.321 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-10T03:33:48.321 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout 0 2024-09-10T03:33:48.321 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout ], 2024-09-10T03:33:48.322 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-10T03:33:48.322 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "smithi112" 2024-09-10T03:33:48.322 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout ], 2024-09-10T03:33:48.322 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-09-10T03:33:48.322 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-10T03:33:48.322 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T03:33:48.322 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-10T03:33:48.322 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-10T03:33:48.322 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:48.322 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-10T03:33:48.322 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T03:33:48.322 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-10T03:33:48.322 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-10T03:33:48.323 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-10T03:33:48.323 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-10T03:33:48.323 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-10T03:33:48.323 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-10T03:33:48.326 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:48.326 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-10T03:33:48.326 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-10T03:33:48.326 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-10T03:33:48.326 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-10T03:33:48.326 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-10T03:33:48.326 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-10T03:33:48.326 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-10T03:33:48.326 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-10T03:33:48.326 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-10T03:33:48.326 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:48.327 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-10T03:33:48.327 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T03:33:48.327 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "btime": "2024-09-10T03:33:31:106056+0000", 2024-09-10T03:33:48.327 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-10T03:33:48.327 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-10T03:33:48.327 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:48.327 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-10T03:33:48.327 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-10T03:33:48.327 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-10T03:33:48.327 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-10T03:33:48.327 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-10T03:33:48.327 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-10T03:33:48.327 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "restful" 2024-09-10T03:33:48.327 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout ], 2024-09-10T03:33:48.327 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T03:33:48.328 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:48.328 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-10T03:33:48.328 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T03:33:48.328 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "modified": "2024-09-10T03:33:31.106680+0000", 2024-09-10T03:33:48.328 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T03:33:48.328 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:48.328 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-10T03:33:48.328 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout } 2024-09-10T03:33:48.328 INFO:teuthology.orchestra.run.smithi112.stdout:mgr not available, waiting (1/15)... 2024-09-10T03:33:51.046 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:50 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/1184400694' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-10T03:33:52.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:51 smithi112 ceph-mon[27737]: Activating manager daemon smithi112.pqrlrb 2024-09-10T03:33:52.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:51 smithi112 ceph-mon[27737]: mgrmap e2: smithi112.pqrlrb(active, starting, since 0.0030253s) 2024-09-10T03:33:52.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:51 smithi112 ceph-mon[27737]: from='mgr.14100 172.21.15.112:0/3346091591' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T03:33:52.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:51 smithi112 ceph-mon[27737]: from='mgr.14100 172.21.15.112:0/3346091591' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T03:33:52.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:51 smithi112 ceph-mon[27737]: from='mgr.14100 172.21.15.112:0/3346091591' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T03:33:52.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:51 smithi112 ceph-mon[27737]: from='mgr.14100 172.21.15.112:0/3346091591' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2024-09-10T03:33:52.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:51 smithi112 ceph-mon[27737]: from='mgr.14100 172.21.15.112:0/3346091591' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr metadata", "who": "smithi112.pqrlrb", "id": "smithi112.pqrlrb"}]: dispatch 2024-09-10T03:33:52.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:51 smithi112 ceph-mon[27737]: Manager daemon smithi112.pqrlrb is now available 2024-09-10T03:33:52.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:51 smithi112 ceph-mon[27737]: from='mgr.14100 172.21.15.112:0/3346091591' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi112.pqrlrb/mirror_snapshot_schedule"}]: dispatch 2024-09-10T03:33:52.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:51 smithi112 ceph-mon[27737]: from='mgr.14100 172.21.15.112:0/3346091591' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi112.pqrlrb/trash_purge_schedule"}]: dispatch 2024-09-10T03:33:52.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:51 smithi112 ceph-mon[27737]: from='mgr.14100 172.21.15.112:0/3346091591' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:33:52.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:51 smithi112 ceph-mon[27737]: from='mgr.14100 172.21.15.112:0/3346091591' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:33:52.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:51 smithi112 ceph-mon[27737]: from='mgr.14100 172.21.15.112:0/3346091591' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:33:52.980 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout 2024-09-10T03:33:52.980 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout { 2024-09-10T03:33:52.980 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "fsid": "513b3bc6-6f25-11ef-bcea-c7b262605968", 2024-09-10T03:33:52.980 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "health": { 2024-09-10T03:33:52.980 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-10T03:33:52.980 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-10T03:33:52.980 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-10T03:33:52.980 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:52.980 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-10T03:33:52.980 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-10T03:33:52.981 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout 0 2024-09-10T03:33:52.981 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout ], 2024-09-10T03:33:52.981 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-10T03:33:52.981 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "smithi112" 2024-09-10T03:33:52.981 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout ], 2024-09-10T03:33:52.981 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-10T03:33:52.981 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-10T03:33:52.981 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T03:33:52.981 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-10T03:33:52.981 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-10T03:33:52.982 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:52.982 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-10T03:33:52.982 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T03:33:52.982 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-10T03:33:52.982 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-10T03:33:52.982 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-10T03:33:52.982 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-10T03:33:52.982 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-10T03:33:52.982 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-10T03:33:52.982 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:52.982 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-10T03:33:52.982 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-10T03:33:52.982 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-10T03:33:52.983 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-10T03:33:52.983 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-10T03:33:52.983 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-10T03:33:52.983 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-10T03:33:52.983 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-10T03:33:52.983 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-10T03:33:52.983 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:52.983 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-10T03:33:52.983 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T03:33:52.983 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "btime": "2024-09-10T03:33:31:106056+0000", 2024-09-10T03:33:52.983 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-10T03:33:52.983 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-10T03:33:52.983 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:52.983 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-10T03:33:52.984 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-10T03:33:52.984 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-10T03:33:52.984 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-10T03:33:52.984 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-10T03:33:52.984 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-10T03:33:52.984 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "restful" 2024-09-10T03:33:52.984 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout ], 2024-09-10T03:33:52.984 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T03:33:52.984 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:52.984 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-10T03:33:52.985 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T03:33:52.985 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "modified": "2024-09-10T03:33:31.106680+0000", 2024-09-10T03:33:52.985 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T03:33:52.985 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:52.985 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-10T03:33:52.985 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout } 2024-09-10T03:33:52.985 INFO:teuthology.orchestra.run.smithi112.stdout:mgr not available, waiting (2/15)... 2024-09-10T03:33:54.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:53 smithi112 ceph-mon[27737]: mgrmap e3: smithi112.pqrlrb(active, since 1.00717s) 2024-09-10T03:33:55.104 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:54 smithi112 ceph-mon[27737]: mgrmap e4: smithi112.pqrlrb(active, since 2s) 2024-09-10T03:33:56.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:55 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/429056105' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-10T03:33:56.241 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout 2024-09-10T03:33:56.241 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout { 2024-09-10T03:33:56.242 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "fsid": "513b3bc6-6f25-11ef-bcea-c7b262605968", 2024-09-10T03:33:56.242 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "health": { 2024-09-10T03:33:56.242 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-10T03:33:56.242 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-10T03:33:56.242 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-10T03:33:56.242 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:56.242 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-10T03:33:56.242 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-10T03:33:56.242 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout 0 2024-09-10T03:33:56.242 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout ], 2024-09-10T03:33:56.242 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-10T03:33:56.242 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "smithi112" 2024-09-10T03:33:56.242 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout ], 2024-09-10T03:33:56.243 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-09-10T03:33:56.243 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-10T03:33:56.243 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T03:33:56.243 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-10T03:33:56.243 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-10T03:33:56.243 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:56.243 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-10T03:33:56.243 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T03:33:56.243 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-10T03:33:56.243 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-10T03:33:56.243 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-10T03:33:56.243 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-10T03:33:56.243 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-10T03:33:56.244 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-10T03:33:56.247 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:56.247 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-10T03:33:56.247 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-10T03:33:56.247 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-10T03:33:56.248 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-10T03:33:56.248 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-10T03:33:56.248 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-10T03:33:56.248 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-10T03:33:56.248 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-10T03:33:56.248 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-10T03:33:56.248 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:56.248 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-10T03:33:56.248 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T03:33:56.248 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "btime": "2024-09-10T03:33:31:106056+0000", 2024-09-10T03:33:56.248 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-10T03:33:56.248 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-10T03:33:56.248 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:56.248 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-10T03:33:56.249 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-10T03:33:56.249 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-10T03:33:56.249 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-10T03:33:56.249 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-10T03:33:56.249 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-10T03:33:56.249 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "restful" 2024-09-10T03:33:56.249 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout ], 2024-09-10T03:33:56.249 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T03:33:56.249 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:56.249 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-10T03:33:56.249 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T03:33:56.249 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "modified": "2024-09-10T03:33:31.106680+0000", 2024-09-10T03:33:56.249 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T03:33:56.250 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout }, 2024-09-10T03:33:56.250 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-10T03:33:56.250 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout } 2024-09-10T03:33:56.250 INFO:teuthology.orchestra.run.smithi112.stdout:mgr is available 2024-09-10T03:33:57.167 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:56 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3567197838' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-10T03:33:57.481 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout 2024-09-10T03:33:57.481 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout [global] 2024-09-10T03:33:57.481 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout fsid = 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:33:57.481 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.112:3300,v1:172.21.15.112:6789] 2024-09-10T03:33:57.481 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-10T03:33:57.481 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-10T03:33:57.482 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-10T03:33:57.482 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-10T03:33:57.482 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout 2024-09-10T03:33:57.482 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-10T03:33:57.482 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-10T03:33:57.482 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout 2024-09-10T03:33:57.482 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout [osd] 2024-09-10T03:33:57.482 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-10T03:33:57.482 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-10T03:33:57.482 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-10T03:33:57.482 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-10T03:33:57.482 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-10T03:33:57.483 INFO:teuthology.orchestra.run.smithi112.stdout:Enabling cephadm module... 2024-09-10T03:33:58.517 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:58 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2752306057' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-10T03:33:59.459 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:59 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2752306057' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-10T03:33:59.459 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:33:59 smithi112 ceph-mon[27737]: mgrmap e5: smithi112.pqrlrb(active, since 6s) 2024-09-10T03:34:00.161 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout { 2024-09-10T03:34:00.161 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-10T03:34:00.161 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-10T03:34:00.161 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "active_name": "smithi112.pqrlrb", 2024-09-10T03:34:00.161 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-10T03:34:00.161 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout } 2024-09-10T03:34:00.162 INFO:teuthology.orchestra.run.smithi112.stdout:Waiting for the mgr to restart... 2024-09-10T03:34:00.162 INFO:teuthology.orchestra.run.smithi112.stdout:Waiting for mgr epoch 5... 2024-09-10T03:34:00.442 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:00 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2894415337' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-10T03:34:03.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:03 smithi112 ceph-mon[27737]: Active manager daemon smithi112.pqrlrb restarted 2024-09-10T03:34:03.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:03 smithi112 ceph-mon[27737]: Activating manager daemon smithi112.pqrlrb 2024-09-10T03:34:03.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:03 smithi112 ceph-mon[27737]: osdmap e2: 0 total, 0 up, 0 in 2024-09-10T03:34:03.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:03 smithi112 ceph-mon[27737]: mgrmap e6: smithi112.pqrlrb(active, starting, since 0.0033105s) 2024-09-10T03:34:03.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:03 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2024-09-10T03:34:03.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:03 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr metadata", "who": "smithi112.pqrlrb", "id": "smithi112.pqrlrb"}]: dispatch 2024-09-10T03:34:03.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:03 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T03:34:03.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:03 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T03:34:03.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:03 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T03:34:03.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:03 smithi112 ceph-mon[27737]: Manager daemon smithi112.pqrlrb is now available 2024-09-10T03:34:03.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:03 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:03.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:03 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:03.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:03 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:34:03.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:03 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:34:03.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:03 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi112.pqrlrb/mirror_snapshot_schedule"}]: dispatch 2024-09-10T03:34:03.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:03 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi112.pqrlrb/trash_purge_schedule"}]: dispatch 2024-09-10T03:34:04.321 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:04 smithi112 ceph-mon[27737]: Found migration_current of "None". Setting to last migration. 2024-09-10T03:34:04.321 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:04 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:04.321 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:04 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:04.321 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:04 smithi112 ceph-mon[27737]: mgrmap e7: smithi112.pqrlrb(active, since 1.00498s) 2024-09-10T03:34:04.583 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout { 2024-09-10T03:34:04.583 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-10T03:34:04.583 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-10T03:34:04.583 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout } 2024-09-10T03:34:04.583 INFO:teuthology.orchestra.run.smithi112.stdout:mgr epoch 5 is available 2024-09-10T03:34:04.583 INFO:teuthology.orchestra.run.smithi112.stdout:Setting orchestrator backend to cephadm... 2024-09-10T03:34:05.216 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:05 smithi112 ceph-mon[27737]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-10T03:34:05.216 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:05 smithi112 ceph-mon[27737]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-10T03:34:05.216 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:05 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:34:06.328 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:06 smithi112 ceph-mon[27737]: [10/Sep/2024:03:34:04] ENGINE Bus STARTING 2024-09-10T03:34:06.328 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:06 smithi112 ceph-mon[27737]: [10/Sep/2024:03:34:04] ENGINE Serving on https://172.21.15.112:7150 2024-09-10T03:34:06.328 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:06 smithi112 ceph-mon[27737]: [10/Sep/2024:03:34:04] ENGINE Error in HTTPServer.serve 2024-09-10T03:34:06.328 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Traceback (most recent call last): 2024-09-10T03:34:06.328 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T03:34:06.328 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: self._connections.run(self.expiration_interval) 2024-09-10T03:34:06.328 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T03:34:06.328 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: self._run(expiration_interval) 2024-09-10T03:34:06.328 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T03:34:06.329 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T03:34:06.329 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T03:34:06.329 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T03:34:06.329 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T03:34:06.329 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: s = self.context.wrap_socket( 2024-09-10T03:34:06.329 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T03:34:06.329 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: return self.sslsocket_class._create( 2024-09-10T03:34:06.329 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T03:34:06.329 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: self.do_handshake() 2024-09-10T03:34:06.329 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T03:34:06.329 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: self._sslobj.do_handshake() 2024-09-10T03:34:06.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T03:34:06.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:06 smithi112 ceph-mon[27737]: [10/Sep/2024:03:34:04] ENGINE Serving on http://172.21.15.112:8765 2024-09-10T03:34:06.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:06 smithi112 ceph-mon[27737]: [10/Sep/2024:03:34:04] ENGINE Bus STARTED 2024-09-10T03:34:06.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:06 smithi112 ceph-mon[27737]: mgrmap e8: smithi112.pqrlrb(active, since 2s) 2024-09-10T03:34:06.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:06 smithi112 ceph-mon[27737]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:06.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:06 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:06.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:06 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:34:06.897 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-10T03:34:06.897 INFO:teuthology.orchestra.run.smithi112.stdout:Generating ssh key... 2024-09-10T03:34:07.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:07 smithi112 ceph-mon[27737]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:08.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:08 smithi112 ceph-mon[27737]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:08.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:08 smithi112 ceph-mon[27737]: Generating ssh key... 2024-09-10T03:34:08.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:08 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:08.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:08 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:09.385 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5UW6iFDDsWMiCodXXnOxTLgSoKrUSy+cCyakRbdg6yJJa3B2AK6o/TgL9MPgsnWPpp070s9bqqK79cz7JHJIpWwuMFKqifCg8hWXrT8ywJS0NMrpqiWk0Hv2Js0qCvHBJV3IPfGob8gq7Nres9jpBr1UEFwDKhA+kOHEj/Emot1IQs/L+MPH31l5WKTqsNNMwAQbrJiUOlCFz5z2mHGOEzmAHcA9Rgh5BlsAZwa1sIAWfXUuP09d83pzxSv+QzDgVC3XS86qEuAlh91yeaGl7AsXjXEgmYs1VEjAocvWJVuThlPlW8rZFiuTQctkdtNh4QbRwHaJTqOStlYKmOi95sdyed4taoIR52+NnZeuAcKdigfF51iWROI/XiDARmheK2U/V11g+j3J4l4fkku3oIWEo9PsaCk8Hf9uayGOz22MOcqMzItF7CDw39eyolY16d4wb0LxvdwpTibv7PNblHjhhByX81I5Y2VRb943gEgyAjFpfl5YqoatklR/BySM= ceph-513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:34:09.386 INFO:teuthology.orchestra.run.smithi112.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-10T03:34:09.386 INFO:teuthology.orchestra.run.smithi112.stdout:Adding key to root@localhost authorized_keys... 2024-09-10T03:34:09.392 INFO:teuthology.orchestra.run.smithi112.stdout:Adding host smithi112... 2024-09-10T03:34:10.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:09 smithi112 ceph-mon[27737]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:10.797 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:10 smithi112 ceph-mon[27737]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi112", "addr": "172.21.15.112", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:11.728 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:11 smithi112 ceph-mon[27737]: Deploying cephadm binary to smithi112 2024-09-10T03:34:13.839 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout Added host 'smithi112' with addr '172.21.15.112' 2024-09-10T03:34:13.840 INFO:teuthology.orchestra.run.smithi112.stdout:Deploying mon service with default placement... 2024-09-10T03:34:14.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:14 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:14.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:14 smithi112 ceph-mon[27737]: Added host smithi112 2024-09-10T03:34:14.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:14 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:34:15.599 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-10T03:34:15.599 INFO:teuthology.orchestra.run.smithi112.stdout:Deploying mgr service with default placement... 2024-09-10T03:34:15.714 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:15 smithi112 ceph-mon[27737]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:15.714 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:15 smithi112 ceph-mon[27737]: Saving service mon spec with placement count:5 2024-09-10T03:34:15.714 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:15 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:16.733 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-10T03:34:16.733 INFO:teuthology.orchestra.run.smithi112.stdout:Deploying crash service with default placement... 2024-09-10T03:34:17.300 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:17 smithi112 ceph-mon[27737]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:17.300 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:17 smithi112 ceph-mon[27737]: Saving service mgr spec with placement count:2 2024-09-10T03:34:17.300 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:17 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:17.923 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-09-10T03:34:17.924 INFO:teuthology.orchestra.run.smithi112.stdout:Deploying ceph-exporter service with default placement... 2024-09-10T03:34:18.536 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:18 smithi112 ceph-mon[27737]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:18.536 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:18 smithi112 ceph-mon[27737]: Saving service crash spec with placement * 2024-09-10T03:34:18.536 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:18 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:18.536 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:18 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:18.536 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:18 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:19.190 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-09-10T03:34:19.190 INFO:teuthology.orchestra.run.smithi112.stdout:Deploying prometheus service with default placement... 2024-09-10T03:34:19.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:19 smithi112 ceph-mon[27737]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:19.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:19 smithi112 ceph-mon[27737]: Saving service ceph-exporter spec with placement * 2024-09-10T03:34:19.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:19 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:20.891 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-09-10T03:34:20.891 INFO:teuthology.orchestra.run.smithi112.stdout:Deploying grafana service with default placement... 2024-09-10T03:34:21.159 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:20 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:21.159 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:20 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:21.159 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:20 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:22.075 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-09-10T03:34:22.075 INFO:teuthology.orchestra.run.smithi112.stdout:Deploying node-exporter service with default placement... 2024-09-10T03:34:22.197 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:21 smithi112 ceph-mon[27737]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:22.197 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:21 smithi112 ceph-mon[27737]: Saving service prometheus spec with placement count:1 2024-09-10T03:34:22.197 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:21 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:23.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:22 smithi112 ceph-mon[27737]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:23.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:22 smithi112 ceph-mon[27737]: Saving service grafana spec with placement count:1 2024-09-10T03:34:23.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:22 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:23.853 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-09-10T03:34:23.853 INFO:teuthology.orchestra.run.smithi112.stdout:Deploying alertmanager service with default placement... 2024-09-10T03:34:24.133 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:24 smithi112 ceph-mon[27737]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:24.133 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:24 smithi112 ceph-mon[27737]: Saving service node-exporter spec with placement * 2024-09-10T03:34:24.133 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:24 smithi112 ceph-mon[27737]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:34:25.093 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-09-10T03:34:25.703 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:25 smithi112 ceph-mon[27737]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:25.703 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:25 smithi112 ceph-mon[27737]: Saving service alertmanager spec with placement count:1 2024-09-10T03:34:25.703 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:25 smithi112 ceph-mon[27737]: from='mgr.14118 172.21.15.112:0/2496735746' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:25.703 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:25 smithi112 ceph-mon[27737]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:34:26.916 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:26 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3363694640' entity='client.admin' 2024-09-10T03:34:27.481 INFO:teuthology.orchestra.run.smithi112.stdout:Enabling the dashboard module... 2024-09-10T03:34:28.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:27 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3610632132' entity='client.admin' 2024-09-10T03:34:28.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:27 smithi112 ceph-mon[27737]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:34:29.218 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:28 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/847332775' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-10T03:34:30.017 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:29 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/847332775' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-10T03:34:30.018 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:29 smithi112 ceph-mon[27737]: mgrmap e9: smithi112.pqrlrb(active, since 25s) 2024-09-10T03:34:30.754 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout { 2024-09-10T03:34:30.754 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-10T03:34:30.755 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-10T03:34:30.755 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "active_name": "smithi112.pqrlrb", 2024-09-10T03:34:30.755 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-10T03:34:30.755 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout } 2024-09-10T03:34:30.755 INFO:teuthology.orchestra.run.smithi112.stdout:Waiting for the mgr to restart... 2024-09-10T03:34:30.755 INFO:teuthology.orchestra.run.smithi112.stdout:Waiting for mgr epoch 9... 2024-09-10T03:34:31.011 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:30 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3886696855' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-10T03:34:34.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:33 smithi112 ceph-mon[27737]: Active manager daemon smithi112.pqrlrb restarted 2024-09-10T03:34:34.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:33 smithi112 ceph-mon[27737]: Activating manager daemon smithi112.pqrlrb 2024-09-10T03:34:34.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:33 smithi112 ceph-mon[27737]: osdmap e3: 0 total, 0 up, 0 in 2024-09-10T03:34:34.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:33 smithi112 ceph-mon[27737]: mgrmap e10: smithi112.pqrlrb(active, starting, since 0.0058701s) 2024-09-10T03:34:34.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:33 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2024-09-10T03:34:34.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:33 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr metadata", "who": "smithi112.pqrlrb", "id": "smithi112.pqrlrb"}]: dispatch 2024-09-10T03:34:34.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:33 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T03:34:34.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:33 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T03:34:34.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:33 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T03:34:34.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:33 smithi112 ceph-mon[27737]: Manager daemon smithi112.pqrlrb is now available 2024-09-10T03:34:34.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:33 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:34:34.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:33 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi112.pqrlrb/mirror_snapshot_schedule"}]: dispatch 2024-09-10T03:34:34.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:33 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi112.pqrlrb/trash_purge_schedule"}]: dispatch 2024-09-10T03:34:35.580 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout { 2024-09-10T03:34:35.580 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-10T03:34:35.580 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-10T03:34:35.581 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout } 2024-09-10T03:34:35.581 INFO:teuthology.orchestra.run.smithi112.stdout:mgr epoch 9 is available 2024-09-10T03:34:35.581 INFO:teuthology.orchestra.run.smithi112.stdout:Generating a dashboard self-signed certificate... 2024-09-10T03:34:35.981 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:35 smithi112 ceph-mon[27737]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-10T03:34:35.981 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:35 smithi112 ceph-mon[27737]: mgrmap e11: smithi112.pqrlrb(active, since 1.00572s) 2024-09-10T03:34:35.981 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:35 smithi112 ceph-mon[27737]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-10T03:34:35.981 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:35 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:37.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:36 smithi112 ceph-mon[27737]: [10/Sep/2024:03:34:35] ENGINE Bus STARTING 2024-09-10T03:34:37.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:36 smithi112 ceph-mon[27737]: [10/Sep/2024:03:34:35] ENGINE Serving on https://172.21.15.112:7150 2024-09-10T03:34:37.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:36 smithi112 ceph-mon[27737]: [10/Sep/2024:03:34:35] ENGINE Error in HTTPServer.serve 2024-09-10T03:34:37.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Traceback (most recent call last): 2024-09-10T03:34:37.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T03:34:37.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: self._connections.run(self.expiration_interval) 2024-09-10T03:34:37.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T03:34:37.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: self._run(expiration_interval) 2024-09-10T03:34:37.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T03:34:37.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T03:34:37.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T03:34:37.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T03:34:37.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T03:34:37.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: s = self.context.wrap_socket( 2024-09-10T03:34:37.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T03:34:37.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: return self.sslsocket_class._create( 2024-09-10T03:34:37.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T03:34:37.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: self.do_handshake() 2024-09-10T03:34:37.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T03:34:37.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: self._sslobj.do_handshake() 2024-09-10T03:34:37.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T03:34:37.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:36 smithi112 ceph-mon[27737]: [10/Sep/2024:03:34:35] ENGINE Serving on http://172.21.15.112:8765 2024-09-10T03:34:37.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:36 smithi112 ceph-mon[27737]: [10/Sep/2024:03:34:35] ENGINE Bus STARTED 2024-09-10T03:34:37.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:36 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:37.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:36 smithi112 ceph-mon[27737]: mgrmap e12: smithi112.pqrlrb(active, since 2s) 2024-09-10T03:34:37.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:36 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:37.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:36 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:37.878 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-10T03:34:37.878 INFO:teuthology.orchestra.run.smithi112.stdout:Creating initial admin user... 2024-09-10T03:34:38.003 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:37 smithi112 ceph-mon[27737]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:39.309 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$E0Tn3f59qgDMt57ga44sRuf9YPvOZKUCbHvS9pgsNa6HOfDHXGUCu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1725939278, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-10T03:34:39.309 INFO:teuthology.orchestra.run.smithi112.stdout:Fetching dashboard port number... 2024-09-10T03:34:39.869 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:39 smithi112 ceph-mon[27737]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:39.869 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:39 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:40.466 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stdout 8443 2024-09-10T03:34:40.475 INFO:teuthology.orchestra.run.smithi112.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-10T03:34:40.475 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stdout disabled 2024-09-10T03:34:40.482 INFO:teuthology.orchestra.run.smithi112.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-10T03:34:40.483 INFO:teuthology.orchestra.run.smithi112.stdout:systemctl: stdout inactive 2024-09-10T03:34:40.483 INFO:teuthology.orchestra.run.smithi112.stdout:firewalld.service is not enabled 2024-09-10T03:34:40.483 INFO:teuthology.orchestra.run.smithi112.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-10T03:34:40.484 INFO:teuthology.orchestra.run.smithi112.stdout:Ceph Dashboard is now available at: 2024-09-10T03:34:40.485 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:34:40.485 INFO:teuthology.orchestra.run.smithi112.stdout: URL: https://smithi112.front.sepia.ceph.com:8443/ 2024-09-10T03:34:40.485 INFO:teuthology.orchestra.run.smithi112.stdout: User: admin 2024-09-10T03:34:40.485 INFO:teuthology.orchestra.run.smithi112.stdout: Password: 8lidylkz0l 2024-09-10T03:34:40.485 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:34:40.485 INFO:teuthology.orchestra.run.smithi112.stdout:Saving cluster configuration to /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config directory 2024-09-10T03:34:40.714 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:40 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3145608084' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-10T03:34:41.736 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:41 smithi112 ceph-mon[27737]: mgrmap e13: smithi112.pqrlrb(active, since 6s) 2024-09-10T03:34:41.736 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:41 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2887100355' entity='client.admin' 2024-09-10T03:34:41.738 INFO:teuthology.orchestra.run.smithi112.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-10T03:34:41.739 INFO:teuthology.orchestra.run.smithi112.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-10T03:34:41.739 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:34:41.739 INFO:teuthology.orchestra.run.smithi112.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-10T03:34:41.739 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:34:41.739 INFO:teuthology.orchestra.run.smithi112.stdout:Or, if you are only running a single cluster on this host: 2024-09-10T03:34:41.739 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:34:41.739 INFO:teuthology.orchestra.run.smithi112.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-10T03:34:41.739 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:34:41.739 INFO:teuthology.orchestra.run.smithi112.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-10T03:34:41.739 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:34:41.739 INFO:teuthology.orchestra.run.smithi112.stdout: ceph telemetry on 2024-09-10T03:34:41.740 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:34:41.740 INFO:teuthology.orchestra.run.smithi112.stdout:For more information see: 2024-09-10T03:34:41.740 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:34:41.740 INFO:teuthology.orchestra.run.smithi112.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-10T03:34:41.740 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:34:41.740 INFO:teuthology.orchestra.run.smithi112.stdout:Bootstrap complete. 2024-09-10T03:34:41.776 INFO:tasks.cephadm:Fetching config... 2024-09-10T03:34:41.776 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T03:34:41.777 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-10T03:34:41.792 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-10T03:34:41.793 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T03:34:41.793 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-10T03:34:41.848 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-10T03:34:41.848 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T03:34:41.848 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/keyring of=/dev/stdout 2024-09-10T03:34:41.913 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-10T03:34:41.913 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T03:34:41.913 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-10T03:34:41.969 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-10T03:34:41.969 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5UW6iFDDsWMiCodXXnOxTLgSoKrUSy+cCyakRbdg6yJJa3B2AK6o/TgL9MPgsnWPpp070s9bqqK79cz7JHJIpWwuMFKqifCg8hWXrT8ywJS0NMrpqiWk0Hv2Js0qCvHBJV3IPfGob8gq7Nres9jpBr1UEFwDKhA+kOHEj/Emot1IQs/L+MPH31l5WKTqsNNMwAQbrJiUOlCFz5z2mHGOEzmAHcA9Rgh5BlsAZwa1sIAWfXUuP09d83pzxSv+QzDgVC3XS86qEuAlh91yeaGl7AsXjXEgmYs1VEjAocvWJVuThlPlW8rZFiuTQctkdtNh4QbRwHaJTqOStlYKmOi95sdyed4taoIR52+NnZeuAcKdigfF51iWROI/XiDARmheK2U/V11g+j3J4l4fkku3oIWEo9PsaCk8Hf9uayGOz22MOcqMzItF7CDw39eyolY16d4wb0LxvdwpTibv7PNblHjhhByX81I5Y2VRb943gEgyAjFpfl5YqoatklR/BySM= ceph-513b3bc6-6f25-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-10T03:34:42.042 INFO:teuthology.orchestra.run.smithi112.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5UW6iFDDsWMiCodXXnOxTLgSoKrUSy+cCyakRbdg6yJJa3B2AK6o/TgL9MPgsnWPpp070s9bqqK79cz7JHJIpWwuMFKqifCg8hWXrT8ywJS0NMrpqiWk0Hv2Js0qCvHBJV3IPfGob8gq7Nres9jpBr1UEFwDKhA+kOHEj/Emot1IQs/L+MPH31l5WKTqsNNMwAQbrJiUOlCFz5z2mHGOEzmAHcA9Rgh5BlsAZwa1sIAWfXUuP09d83pzxSv+QzDgVC3XS86qEuAlh91yeaGl7AsXjXEgmYs1VEjAocvWJVuThlPlW8rZFiuTQctkdtNh4QbRwHaJTqOStlYKmOi95sdyed4taoIR52+NnZeuAcKdigfF51iWROI/XiDARmheK2U/V11g+j3J4l4fkku3oIWEo9PsaCk8Hf9uayGOz22MOcqMzItF7CDw39eyolY16d4wb0LxvdwpTibv7PNblHjhhByX81I5Y2VRb943gEgyAjFpfl5YqoatklR/BySM= ceph-513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:34:42.053 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5UW6iFDDsWMiCodXXnOxTLgSoKrUSy+cCyakRbdg6yJJa3B2AK6o/TgL9MPgsnWPpp070s9bqqK79cz7JHJIpWwuMFKqifCg8hWXrT8ywJS0NMrpqiWk0Hv2Js0qCvHBJV3IPfGob8gq7Nres9jpBr1UEFwDKhA+kOHEj/Emot1IQs/L+MPH31l5WKTqsNNMwAQbrJiUOlCFz5z2mHGOEzmAHcA9Rgh5BlsAZwa1sIAWfXUuP09d83pzxSv+QzDgVC3XS86qEuAlh91yeaGl7AsXjXEgmYs1VEjAocvWJVuThlPlW8rZFiuTQctkdtNh4QbRwHaJTqOStlYKmOi95sdyed4taoIR52+NnZeuAcKdigfF51iWROI/XiDARmheK2U/V11g+j3J4l4fkku3oIWEo9PsaCk8Hf9uayGOz22MOcqMzItF7CDw39eyolY16d4wb0LxvdwpTibv7PNblHjhhByX81I5Y2VRb943gEgyAjFpfl5YqoatklR/BySM= ceph-513b3bc6-6f25-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-10T03:34:42.277 INFO:teuthology.orchestra.run.smithi169.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5UW6iFDDsWMiCodXXnOxTLgSoKrUSy+cCyakRbdg6yJJa3B2AK6o/TgL9MPgsnWPpp070s9bqqK79cz7JHJIpWwuMFKqifCg8hWXrT8ywJS0NMrpqiWk0Hv2Js0qCvHBJV3IPfGob8gq7Nres9jpBr1UEFwDKhA+kOHEj/Emot1IQs/L+MPH31l5WKTqsNNMwAQbrJiUOlCFz5z2mHGOEzmAHcA9Rgh5BlsAZwa1sIAWfXUuP09d83pzxSv+QzDgVC3XS86qEuAlh91yeaGl7AsXjXEgmYs1VEjAocvWJVuThlPlW8rZFiuTQctkdtNh4QbRwHaJTqOStlYKmOi95sdyed4taoIR52+NnZeuAcKdigfF51iWROI/XiDARmheK2U/V11g+j3J4l4fkku3oIWEo9PsaCk8Hf9uayGOz22MOcqMzItF7CDw39eyolY16d4wb0LxvdwpTibv7PNblHjhhByX81I5Y2VRb943gEgyAjFpfl5YqoatklR/BySM= ceph-513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:34:42.292 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-10T03:34:42.536 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:34:44.174 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-10T03:34:44.174 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-10T03:34:44.473 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:44 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/452510512' entity='client.admin' 2024-09-10T03:34:44.628 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:34:46.457 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi169 2024-09-10T03:34:46.458 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-10T03:34:46.458 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/etc/ceph/ceph.conf 2024-09-10T03:34:46.478 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-10T03:34:46.478 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-10T03:34:46.536 INFO:tasks.cephadm:Adding host smithi169 to orchestrator... 2024-09-10T03:34:46.536 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph orch host add smithi169 2024-09-10T03:34:46.558 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:46 smithi112 ceph-mon[27737]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:46.558 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:46 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:46.559 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:46 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:46.559 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:46 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:46.559 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:46 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:34:46.559 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:46 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:34:46.559 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:46 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:34:46.559 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:46 smithi112 ceph-mon[27737]: Updating smithi112:/etc/ceph/ceph.conf 2024-09-10T03:34:46.790 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:34:48.062 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:47 smithi112 ceph-mon[27737]: Updating smithi112:/var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T03:34:48.062 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:47 smithi112 ceph-mon[27737]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-09-10T03:34:48.062 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:47 smithi112 ceph-mon[27737]: Updating smithi112:/var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T03:34:48.062 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:47 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:48.062 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:47 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:48.062 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:47 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:48.062 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:47 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi112", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-10T03:34:48.062 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:47 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi112", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-09-10T03:34:48.062 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:47 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:34:48.063 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:47 smithi112 ceph-mon[27737]: Deploying daemon ceph-exporter.smithi112 on smithi112 2024-09-10T03:34:48.970 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:48 smithi112 ceph-mon[27737]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi169", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:50.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:49 smithi112 ceph-mon[27737]: Deploying cephadm binary to smithi169 2024-09-10T03:34:50.701 INFO:teuthology.orchestra.run.smithi112.stdout:Added host 'smithi169' with addr '172.21.15.169' 2024-09-10T03:34:51.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:50 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:51.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:50 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:51.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:50 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:51.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:50 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:51.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:50 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi112", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-10T03:34:51.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:50 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi112", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-09-10T03:34:51.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:50 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:34:51.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:50 smithi112 ceph-mon[27737]: Deploying daemon crash.smithi112 on smithi112 2024-09-10T03:34:51.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:50 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:51.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:50 smithi112 ceph-mon[27737]: Added host smithi169 2024-09-10T03:34:51.469 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph orch host ls --format=json 2024-09-10T03:34:51.931 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:34:52.688 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:34:52.689 INFO:teuthology.orchestra.run.smithi112.stdout:[{"addr": "172.21.15.112", "hostname": "smithi112", "labels": [], "status": ""}, {"addr": "172.21.15.169", "hostname": "smithi169", "labels": [], "status": ""}] 2024-09-10T03:34:53.311 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:53 smithi112 ceph-mon[27737]: from='client.14189 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T03:34:53.338 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-10T03:34:53.338 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd crush tunables default 2024-09-10T03:34:53.772 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:34:54.969 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:54 smithi112 ceph-mon[27737]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:34:54.969 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:54 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:54.969 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:54 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:54.970 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:54 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:54.970 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:54 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:54.970 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:54 smithi112 ceph-mon[27737]: Deploying daemon node-exporter.smithi112 on smithi112 2024-09-10T03:34:54.970 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:54 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:54.970 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:54 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3550454350' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-10T03:34:55.034 INFO:teuthology.orchestra.run.smithi112.stderr:adjusted tunables profile to default 2024-09-10T03:34:55.677 INFO:tasks.cephadm:Adding mon.smithi112 on smithi112 2024-09-10T03:34:55.677 INFO:tasks.cephadm:Adding mon.smithi169 on smithi169 2024-09-10T03:34:55.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph orch apply mon '2;smithi112:172.21.15.112=smithi112;smithi169:172.21.15.169=smithi169' 2024-09-10T03:34:55.923 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:34:55.969 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:34:56.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:56 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3550454350' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-10T03:34:56.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:56 smithi112 ceph-mon[27737]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T03:34:56.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:56 smithi112 ceph-mon[27737]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:34:56.584 INFO:teuthology.orchestra.run.smithi169.stdout:Scheduled mon update... 2024-09-10T03:34:57.108 DEBUG:teuthology.orchestra.run.smithi169:mon.smithi169> sudo journalctl -f -n 0 -u ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@mon.smithi169.service 2024-09-10T03:34:57.111 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:34:57.112 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:34:57.383 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:34:57.428 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:34:58.050 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:34:58.050 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:34:58.050 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:34:58.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:57 smithi112 ceph-mon[27737]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi112:172.21.15.112=smithi112;smithi169:172.21.15.169=smithi169", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:34:58.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:57 smithi112 ceph-mon[27737]: Saving service mon spec with placement smithi112:172.21.15.112=smithi112;smithi169:172.21.15.169=smithi169;count:2 2024-09-10T03:34:58.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:57 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:59.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:58 smithi112 ceph-mon[27737]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:34:59.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:58 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:59.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:58 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:59.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:58 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:59.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:58 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:34:59.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:58 smithi112 ceph-mon[27737]: Deploying daemon alertmanager.smithi112 on smithi112 2024-09-10T03:34:59.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:58 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/423084877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:34:59.623 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:34:59.624 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:34:59.872 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:34:59.917 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:00.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:34:59 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:00.521 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:00.521 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:00.522 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:01.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:00 smithi112 ceph-mon[27737]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:01.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:00 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/2140320670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:02.208 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:02.209 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:02.449 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:02.502 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:03.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:02 smithi112 ceph-mon[27737]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:03.189 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:03.189 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:03.189 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:03.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:03 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/3103075283' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:04.768 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:04.768 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:04.955 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:04 smithi112 ceph-mon[27737]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:05.014 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:05.058 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:05.744 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:05.744 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:05.744 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:06.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:05 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/1105217764' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:07.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:06 smithi112 ceph-mon[27737]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:07.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:06 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:07.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:06 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:07.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:06 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:07.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:06 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:07.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:06 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:07.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:06 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:07.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:06 smithi112 ceph-mon[27737]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-10T03:35:07.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:06 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:07.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:06 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:07.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:06 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T03:35:07.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:06 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:07.342 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:07.343 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:07.589 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:07.633 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:08.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:07 smithi112 ceph-mon[27737]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T03:35:08.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:07 smithi112 ceph-mon[27737]: Deploying daemon grafana.smithi112 on smithi112 2024-09-10T03:35:08.436 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:08.436 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:08.437 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:09.249 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:08 smithi112 ceph-mon[27737]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:09.249 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:08 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/1051871552' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:09.249 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:08 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:09.956 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:09.957 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:10.200 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:10.245 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:10.941 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:10.941 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:10.941 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:11.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:10 smithi112 ceph-mon[27737]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:12.231 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:11 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/1158040240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:12.666 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:12.666 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:12.903 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:12.948 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:13.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:12 smithi112 ceph-mon[27737]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:13.621 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:13.622 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:13.622 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:14.030 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:13 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/540773876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:14.935 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:14 smithi112 ceph-mon[27737]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:15.254 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:15.254 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:15.501 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:15.546 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:16.338 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:16.338 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:16.338 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:17.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:16 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:17.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:16 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:17.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:16 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:17.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:16 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:17.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:16 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:17.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:16 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:17.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:16 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:17.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:16 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:17.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:16 smithi112 ceph-mon[27737]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:17.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:16 smithi112 ceph-mon[27737]: Deploying daemon prometheus.smithi112 on smithi112 2024-09-10T03:35:17.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:16 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/944806810' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:17.935 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:17.936 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:18.180 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:18.224 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:18.859 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:18.859 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:18.860 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:19.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:18 smithi112 ceph-mon[27737]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:19.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:18 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:20.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:19 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/3695935674' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:20.390 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:20.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:20.634 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:20.680 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:21.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:20 smithi112 ceph-mon[27737]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:21.348 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:21.348 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:21.348 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:22.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:21 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/2692001257' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:22.880 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:22.884 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:23.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:22 smithi112 ceph-mon[27737]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:23.123 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:23.169 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:23.841 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:23.841 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:23.841 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:25.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:24 smithi112 ceph-mon[27737]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:25.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:24 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/2810367880' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:25.421 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:25.421 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:25.667 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:25.712 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:26.630 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:26.630 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:26.631 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:26.897 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:26 smithi112 ceph-mon[27737]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:26.898 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:26 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/2354773801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:28.219 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:28.220 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:28.463 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:28.508 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:28.944 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:28 smithi112 ceph-mon[27737]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:29.126 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:29.126 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:29.126 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:30.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:30 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:30.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:30 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:30.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:30 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:30.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:30 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-10T03:35:30.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:30 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/3753144255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:30.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:30 smithi112 ceph-mon[27737]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:30.691 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:30.692 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:30.937 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:30.982 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:31.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:31 smithi112 ceph-mon[27737]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-10T03:35:31.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:31 smithi112 ceph-mon[27737]: mgrmap e14: smithi112.pqrlrb(active, since 56s) 2024-09-10T03:35:31.661 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:31.662 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:31.662 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:32.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:32 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/3760111480' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:33.215 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:33.215 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:33.459 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:33.504 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:34.148 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:34.149 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:34.149 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:34.295 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:34 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/1859984990' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:35.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:35 smithi112 ceph-mon[27737]: Active manager daemon smithi112.pqrlrb restarted 2024-09-10T03:35:35.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:35 smithi112 ceph-mon[27737]: Activating manager daemon smithi112.pqrlrb 2024-09-10T03:35:35.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:35 smithi112 ceph-mon[27737]: osdmap e5: 0 total, 0 up, 0 in 2024-09-10T03:35:35.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:35 smithi112 ceph-mon[27737]: mgrmap e15: smithi112.pqrlrb(active, starting, since 0.00330035s) 2024-09-10T03:35:35.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2024-09-10T03:35:35.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr metadata", "who": "smithi112.pqrlrb", "id": "smithi112.pqrlrb"}]: dispatch 2024-09-10T03:35:35.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T03:35:35.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T03:35:35.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T03:35:35.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:35 smithi112 ceph-mon[27737]: Manager daemon smithi112.pqrlrb is now available 2024-09-10T03:35:35.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:35:35.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:35.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:35:35.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi112.pqrlrb/mirror_snapshot_schedule"}]: dispatch 2024-09-10T03:35:35.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi112.pqrlrb/trash_purge_schedule"}]: dispatch 2024-09-10T03:35:35.747 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:35.748 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:35.997 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:36.043 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:36.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:36 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:36.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:36 smithi112 ceph-mon[27737]: mgrmap e16: smithi112.pqrlrb(active, since 1.00626s) 2024-09-10T03:35:36.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:36 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:36.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:36 smithi112 ceph-mon[27737]: [10/Sep/2024:03:35:36] ENGINE Bus STARTING 2024-09-10T03:35:36.731 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:36.732 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:36.732 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:37.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:37 smithi112 ceph-mon[27737]: [10/Sep/2024:03:35:36] ENGINE Serving on http://172.21.15.112:8765 2024-09-10T03:35:37.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:37 smithi112 ceph-mon[27737]: [10/Sep/2024:03:35:36] ENGINE Serving on https://172.21.15.112:7150 2024-09-10T03:35:37.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:37 smithi112 ceph-mon[27737]: [10/Sep/2024:03:35:36] ENGINE Bus STARTED 2024-09-10T03:35:37.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:37 smithi112 ceph-mon[27737]: [10/Sep/2024:03:35:36] ENGINE Error in HTTPServer.serve 2024-09-10T03:35:37.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: Traceback (most recent call last): 2024-09-10T03:35:37.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T03:35:37.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: self._connections.run(self.expiration_interval) 2024-09-10T03:35:37.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T03:35:37.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: self._run(expiration_interval) 2024-09-10T03:35:37.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T03:35:37.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T03:35:37.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T03:35:37.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T03:35:37.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T03:35:37.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: s = self.context.wrap_socket( 2024-09-10T03:35:37.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T03:35:37.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: return self.sslsocket_class._create( 2024-09-10T03:35:37.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T03:35:37.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: self.do_handshake() 2024-09-10T03:35:37.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T03:35:37.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: self._sslobj.do_handshake() 2024-09-10T03:35:37.832 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T03:35:37.832 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:37 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/1572000195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:37.832 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:37 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:38.328 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:38.328 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:38.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:38 smithi112 ceph-mon[27737]: mgrmap e17: smithi112.pqrlrb(active, since 2s) 2024-09-10T03:35:38.880 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:38.931 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-10T03:35:39.661 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:39.661 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:39.661 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:40.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:40.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:40.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:40 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/1615686960' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:40.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:40.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:40.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:35:40.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:40.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:40.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:40.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:40.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd/host:smithi169", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:35:40.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:35:40.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:35:40.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:40 smithi112 ceph-mon[27737]: Updating smithi112:/etc/ceph/ceph.conf 2024-09-10T03:35:40.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:40 smithi112 ceph-mon[27737]: Updating smithi169:/etc/ceph/ceph.conf 2024-09-10T03:35:41.593 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:41.594 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:42.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:41 smithi112 ceph-mon[27737]: Updating smithi112:/var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T03:35:42.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:41 smithi112 ceph-mon[27737]: Updating smithi169:/var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T03:35:42.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:41 smithi112 ceph-mon[27737]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-09-10T03:35:42.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:41 smithi112 ceph-mon[27737]: Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2024-09-10T03:35:42.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:41 smithi112 ceph-mon[27737]: Updating smithi112:/var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T03:35:42.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:41 smithi112 ceph-mon[27737]: Updating smithi169:/var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T03:35:42.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:41 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:42.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:41 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:42.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:41 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:42.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:41 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:42.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:41 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:42.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:41 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi169", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-10T03:35:42.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:41 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi169", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-09-10T03:35:42.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:41 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:35:42.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:41 smithi112 ceph-mon[27737]: Deploying daemon ceph-exporter.smithi169 on smithi169 2024-09-10T03:35:42.158 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T03:35:42.986 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:42.986 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:42.986 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:43.295 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:43 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/3298124962' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:44.501 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:44.501 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:45.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:45 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:45.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:45 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:45.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:45 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:45.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:45 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:45.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:45 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi169", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-10T03:35:45.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:45 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi169", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-09-10T03:35:45.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:45 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:35:45.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:45 smithi112 ceph-mon[27737]: Deploying daemon crash.smithi169 on smithi169 2024-09-10T03:35:45.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:45 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:45.388 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T03:35:46.160 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:46.160 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:46.160 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:46.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:46 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/1191909969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:47.802 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:47.802 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:48.088 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T03:35:48.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:48 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:48.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:48 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:48.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:48 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:48.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:48 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:48.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:48 smithi112 ceph-mon[27737]: Deploying daemon node-exporter.smithi169 on smithi169 2024-09-10T03:35:48.798 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:48.798 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:48.798 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:49.323 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:49 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/1541531452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:51.055 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:51.055 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:51.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:50 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:51.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:50 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:35:52.309 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T03:35:52.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:51 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:52.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:51 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:52.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:51 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:52.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:51 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:52.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:51 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi169.bkwgjl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T03:35:52.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:51 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi169.bkwgjl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-10T03:35:52.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:51 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T03:35:52.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:51 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:35:52.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:51 smithi112 ceph-mon[27737]: Deploying daemon mgr.smithi169.bkwgjl on smithi169 2024-09-10T03:35:53.179 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:53.179 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:53.179 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:53.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:53 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/3228768144' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:54.754 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:54.754 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:55.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:55 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:55.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:55 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:55.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:55 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:55.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:55 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:55.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:55 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T03:35:55.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:55 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:35:55.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:55 smithi112 ceph-mon[27737]: Deploying daemon mon.smithi169 on smithi169 2024-09-10T03:35:55.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:55 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:56.150 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T03:35:56.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:35:56 smithi112 ceph-mon[27737]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.296 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:35:57.296 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:33:28.238100Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T03:35:57.296 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2024-09-10T03:35:57.691 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 podman[35637]: 2024-09-10 03:35:57.690713937 +0000 UTC m=+0.528679607 container init 38476c275eb5c2bb3e740552272aa1188a03e6e1d8775e55292d039a92bb9487 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi169, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-10T03:35:57.954 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 podman[35637]: 2024-09-10 03:35:57.694040949 +0000 UTC m=+0.532006606 container start 38476c275eb5c2bb3e740552272aa1188a03e6e1d8775e55292d039a92bb9487 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi169, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-b53b6d3, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-10T03:35:57.954 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: set uid:gid to 167:167 (ceph:ceph) 2024-09-10T03:35:57.954 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable), process ceph-mon, pid 2 2024-09-10T03:35:57.954 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pidfile_write: ignore empty --pid-file 2024-09-10T03:35:57.954 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: load: jerasure load: lrc 2024-09-10T03:35:57.954 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: RocksDB version: 7.9.2 2024-09-10T03:35:57.954 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Git sha 0 2024-09-10T03:35:57.954 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Compile date 2024-09-09 16:23:28 2024-09-10T03:35:57.955 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: DB SUMMARY 2024-09-10T03:35:57.955 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: DB Session ID: EZO4Y18IPRMI6AP3JNMR 2024-09-10T03:35:57.955 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: CURRENT file: CURRENT 2024-09-10T03:35:57.955 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: IDENTITY file: IDENTITY 2024-09-10T03:35:57.955 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-10T03:35:57.955 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi169/store.db dir, Total Num: 0, files: 2024-09-10T03:35:57.955 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi169/store.db: 000004.log size: 511 ; 2024-09-10T03:35:57.955 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.error_if_exists: 0 2024-09-10T03:35:57.955 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.create_if_missing: 0 2024-09-10T03:35:57.955 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.paranoid_checks: 1 2024-09-10T03:35:57.955 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-10T03:35:57.955 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-10T03:35:57.955 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-10T03:35:57.955 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.env: 0x55572e059940 2024-09-10T03:35:57.956 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.fs: PosixFileSystem 2024-09-10T03:35:57.956 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.info_log: 0x55572fcd98c0 2024-09-10T03:35:57.956 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_file_opening_threads: 16 2024-09-10T03:35:57.956 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.statistics: (nil) 2024-09-10T03:35:57.956 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.use_fsync: 0 2024-09-10T03:35:57.956 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_log_file_size: 0 2024-09-10T03:35:57.956 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-10T03:35:57.956 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-10T03:35:57.956 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.keep_log_file_num: 1000 2024-09-10T03:35:57.956 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.recycle_log_file_num: 0 2024-09-10T03:35:57.956 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.allow_fallocate: 1 2024-09-10T03:35:57.959 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.allow_mmap_reads: 0 2024-09-10T03:35:57.959 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.allow_mmap_writes: 0 2024-09-10T03:35:57.960 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.use_direct_reads: 0 2024-09-10T03:35:57.960 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-10T03:35:57.960 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.create_missing_column_families: 0 2024-09-10T03:35:57.960 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.db_log_dir: 2024-09-10T03:35:57.960 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.wal_dir: 2024-09-10T03:35:57.960 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-10T03:35:57.960 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-10T03:35:57.960 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-10T03:35:57.960 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-10T03:35:57.960 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-10T03:35:57.960 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-10T03:35:57.960 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.advise_random_on_open: 1 2024-09-10T03:35:57.960 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.db_write_buffer_size: 0 2024-09-10T03:35:57.961 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.write_buffer_manager: 0x55572fce8dc0 2024-09-10T03:35:57.961 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-10T03:35:57.961 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-10T03:35:57.961 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-10T03:35:57.961 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.rate_limiter: (nil) 2024-09-10T03:35:57.961 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-10T03:35:57.961 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.wal_recovery_mode: 2 2024-09-10T03:35:57.962 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.enable_thread_tracking: 0 2024-09-10T03:35:57.962 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.enable_pipelined_write: 0 2024-09-10T03:35:57.962 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.unordered_write: 0 2024-09-10T03:35:57.962 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-10T03:35:57.962 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-10T03:35:57.962 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-10T03:35:57.962 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-10T03:35:57.962 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.row_cache: None 2024-09-10T03:35:57.962 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.wal_filter: None 2024-09-10T03:35:57.962 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-10T03:35:57.962 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.allow_ingest_behind: 0 2024-09-10T03:35:57.962 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.two_write_queues: 0 2024-09-10T03:35:57.962 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.manual_wal_flush: 0 2024-09-10T03:35:57.963 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.wal_compression: 0 2024-09-10T03:35:57.963 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.atomic_flush: 0 2024-09-10T03:35:57.963 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-10T03:35:57.963 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-10T03:35:57.963 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-10T03:35:57.963 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.log_readahead_size: 0 2024-09-10T03:35:57.963 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-10T03:35:57.963 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.best_efforts_recovery: 0 2024-09-10T03:35:57.963 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-10T03:35:57.963 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-10T03:35:57.963 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.allow_data_in_errors: 0 2024-09-10T03:35:57.963 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.db_host_id: __hostname__ 2024-09-10T03:35:57.963 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-10T03:35:57.964 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_background_jobs: 2 2024-09-10T03:35:57.964 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_background_compactions: -1 2024-09-10T03:35:57.964 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_subcompactions: 1 2024-09-10T03:35:57.964 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-10T03:35:57.964 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-10T03:35:57.964 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-10T03:35:57.964 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_total_wal_size: 0 2024-09-10T03:35:57.964 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-10T03:35:57.964 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-10T03:35:57.964 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-10T03:35:57.964 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-10T03:35:57.964 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_open_files: -1 2024-09-10T03:35:57.964 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.bytes_per_sync: 0 2024-09-10T03:35:57.964 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-10T03:35:57.965 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-10T03:35:57.965 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compaction_readahead_size: 0 2024-09-10T03:35:57.965 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_background_flushes: -1 2024-09-10T03:35:57.965 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Compression algorithms supported: 2024-09-10T03:35:57.965 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: kZSTD supported: 0 2024-09-10T03:35:57.965 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: kXpressCompression supported: 0 2024-09-10T03:35:57.965 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: kBZip2Compression supported: 0 2024-09-10T03:35:57.965 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-10T03:35:57.965 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: kLZ4Compression supported: 1 2024-09-10T03:35:57.965 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: kZlibCompression supported: 1 2024-09-10T03:35:57.965 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: kLZ4HCCompression supported: 1 2024-09-10T03:35:57.966 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: kSnappyCompression supported: 1 2024-09-10T03:35:57.966 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-10T03:35:57.966 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-10T03:35:57.966 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi169/store.db/MANIFEST-000005 2024-09-10T03:35:57.966 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-10T03:35:57.966 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-10T03:35:57.966 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.merge_operator: 2024-09-10T03:35:57.966 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compaction_filter: None 2024-09-10T03:35:57.966 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compaction_filter_factory: None 2024-09-10T03:35:57.966 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.sst_partitioner_factory: None 2024-09-10T03:35:57.966 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-10T03:35:57.967 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-10T03:35:57.967 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55572fcd9520) 2024-09-10T03:35:57.967 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: cache_index_and_filter_blocks: 1 2024-09-10T03:35:57.967 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-10T03:35:57.967 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-10T03:35:57.967 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: pin_top_level_index_and_filter: 1 2024-09-10T03:35:57.967 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: index_type: 0 2024-09-10T03:35:57.967 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: data_block_index_type: 0 2024-09-10T03:35:57.967 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: index_shortening: 1 2024-09-10T03:35:57.967 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-10T03:35:57.967 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: checksum: 4 2024-09-10T03:35:57.967 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: no_block_cache: 0 2024-09-10T03:35:57.967 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: block_cache: 0x55572fccd1f0 2024-09-10T03:35:57.967 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: block_cache_name: BinnedLRUCache 2024-09-10T03:35:57.968 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: block_cache_options: 2024-09-10T03:35:57.968 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: capacity : 536870912 2024-09-10T03:35:57.968 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: num_shard_bits : 4 2024-09-10T03:35:57.968 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: strict_capacity_limit : 0 2024-09-10T03:35:57.968 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: high_pri_pool_ratio: 0.000 2024-09-10T03:35:57.968 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: block_cache_compressed: (nil) 2024-09-10T03:35:57.968 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: persistent_cache: (nil) 2024-09-10T03:35:57.968 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: block_size: 4096 2024-09-10T03:35:57.968 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: block_size_deviation: 10 2024-09-10T03:35:57.968 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: block_restart_interval: 16 2024-09-10T03:35:57.968 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: index_block_restart_interval: 1 2024-09-10T03:35:57.968 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: metadata_block_size: 4096 2024-09-10T03:35:57.968 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: partition_filters: 0 2024-09-10T03:35:57.968 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: use_delta_encoding: 1 2024-09-10T03:35:57.969 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: filter_policy: bloomfilter 2024-09-10T03:35:57.969 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: whole_key_filtering: 1 2024-09-10T03:35:57.969 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: verify_compression: 0 2024-09-10T03:35:57.969 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: read_amp_bytes_per_bit: 0 2024-09-10T03:35:57.969 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: format_version: 5 2024-09-10T03:35:57.969 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: enable_index_compression: 1 2024-09-10T03:35:57.969 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: block_align: 0 2024-09-10T03:35:57.969 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: max_auto_readahead_size: 262144 2024-09-10T03:35:57.969 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: prepopulate_block_cache: 0 2024-09-10T03:35:57.969 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: initial_auto_readahead_size: 8192 2024-09-10T03:35:57.969 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: num_file_reads_for_auto_readahead: 2 2024-09-10T03:35:57.969 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.write_buffer_size: 33554432 2024-09-10T03:35:57.969 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_write_buffer_number: 2 2024-09-10T03:35:57.969 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compression: NoCompression 2024-09-10T03:35:57.970 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.bottommost_compression: Disabled 2024-09-10T03:35:57.970 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.prefix_extractor: nullptr 2024-09-10T03:35:57.970 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-10T03:35:57.970 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.num_levels: 7 2024-09-10T03:35:57.970 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-10T03:35:57.970 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-10T03:35:57.970 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-10T03:35:57.970 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-10T03:35:57.970 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-10T03:35:57.970 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-10T03:35:57.970 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-10T03:35:57.970 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-10T03:35:57.970 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-10T03:35:57.970 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-10T03:35:57.970 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-10T03:35:57.971 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-10T03:35:57.971 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-10T03:35:57.971 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compression_opts.level: 32767 2024-09-10T03:35:57.971 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compression_opts.strategy: 0 2024-09-10T03:35:57.971 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-10T03:35:57.971 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-10T03:35:57.971 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-10T03:35:57.971 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-10T03:35:57.971 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compression_opts.enabled: false 2024-09-10T03:35:57.972 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-10T03:35:57.972 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-10T03:35:57.972 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-10T03:35:57.972 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-10T03:35:57.972 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.target_file_size_base: 67108864 2024-09-10T03:35:57.972 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-10T03:35:57.972 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-10T03:35:57.972 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-10T03:35:57.972 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-10T03:35:57.972 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-10T03:35:57.972 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-10T03:35:57.972 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-10T03:35:57.972 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-10T03:35:57.973 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-10T03:35:57.973 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-10T03:35:57.973 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-10T03:35:57.973 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-10T03:35:57.973 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-10T03:35:57.973 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-10T03:35:57.973 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.arena_block_size: 1048576 2024-09-10T03:35:57.973 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-10T03:35:57.973 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-10T03:35:57.973 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.disable_auto_compactions: 0 2024-09-10T03:35:57.973 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-10T03:35:57.973 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-10T03:35:57.973 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-10T03:35:57.974 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-10T03:35:57.974 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-10T03:35:57.974 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-10T03:35:57.974 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-10T03:35:57.974 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-10T03:35:57.974 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-10T03:35:57.974 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-10T03:35:57.974 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-10T03:35:57.974 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.inplace_update_support: 0 2024-09-10T03:35:57.974 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-10T03:35:57.974 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-10T03:35:57.974 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-10T03:35:57.974 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-10T03:35:57.974 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.bloom_locality: 0 2024-09-10T03:35:57.974 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.max_successive_merges: 0 2024-09-10T03:35:57.975 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-10T03:35:57.975 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.paranoid_file_checks: 0 2024-09-10T03:35:57.975 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.force_consistency_checks: 1 2024-09-10T03:35:57.975 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.report_bg_io_stats: 0 2024-09-10T03:35:57.975 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.ttl: 2592000 2024-09-10T03:35:57.975 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-10T03:35:57.975 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-10T03:35:57.975 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-10T03:35:57.975 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.enable_blob_files: false 2024-09-10T03:35:57.975 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.min_blob_size: 0 2024-09-10T03:35:57.975 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.blob_file_size: 268435456 2024-09-10T03:35:57.975 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-10T03:35:57.975 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-10T03:35:57.975 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-10T03:35:57.976 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-10T03:35:57.976 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-10T03:35:57.976 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.blob_file_starting_level: 0 2024-09-10T03:35:57.976 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-10T03:35:57.976 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi169/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-10T03:35:57.976 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-10T03:35:57.976 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ee508e5a-9b91-4fa9-8a80-630058f1f0f7 2024-09-10T03:35:57.976 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725939357721299, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-10T03:35:57.976 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-10T03:35:57.976 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725939357721876, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1725939357, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ee508e5a-9b91-4fa9-8a80-630058f1f0f7", "db_session_id": "EZO4Y18IPRMI6AP3JNMR", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-10T03:35:57.976 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725939357722001, "job": 1, "event": "recovery_finished"} 2024-09-10T03:35:57.976 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-10T03:35:57.976 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi169/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-10T03:35:57.976 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55572fcfae00 2024-09-10T03:35:57.977 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: DB pointer 0x55572fdf6000 2024-09-10T03:35:57.977 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-10T03:35:57.977 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-10T03:35:57.977 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: ** DB Stats ** 2024-09-10T03:35:57.977 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T03:35:57.977 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-10T03:35:57.977 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-10T03:35:57.977 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T03:35:57.977 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-10T03:35:57.977 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-10T03:35:57.977 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T03:35:57.977 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: 2024-09-10T03:35:57.978 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: ** Compaction Stats [default] ** 2024-09-10T03:35:57.978 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-10T03:35:57.978 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-10T03:35:57.978 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-10T03:35:57.978 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-10T03:35:57.978 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-10T03:35:57.978 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: 2024-09-10T03:35:57.978 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: ** Compaction Stats [default] ** 2024-09-10T03:35:57.978 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-10T03:35:57.978 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T03:35:57.978 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-10T03:35:57.978 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: 2024-09-10T03:35:57.978 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-10T03:35:57.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: 2024-09-10T03:35:57.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T03:35:57.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T03:35:57.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T03:35:57.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T03:35:57.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T03:35:57.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-10T03:35:57.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Cumulative compaction: 0.00 GB write, 0.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T03:35:57.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Interval compaction: 0.00 GB write, 0.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T03:35:57.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T03:35:57.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Block cache BinnedLRUCache@0x55572fccd1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-09-10T03:35:57.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-10T03:35:57.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: 2024-09-10T03:35:57.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-10T03:35:57.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mon.smithi169 does not exist in monmap, will attempt to join an existing cluster 2024-09-10T03:35:57.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: using public_addr v2:172.21.15.169:0/0 -> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] 2024-09-10T03:35:57.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: starting mon.smithi169 rank -1 at public addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] at bind addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi169 fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:35:57.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mon.smithi169@-1(???) e0 preinit fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:35:57.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mon.smithi169@-1(synchronizing).mds e1 new map 2024-09-10T03:35:57.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mon.smithi169@-1(synchronizing).mds e1 print_map 2024-09-10T03:35:57.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: e1 2024-09-10T03:35:57.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: btime 2024-09-10T03:33:31:106056+0000 2024-09-10T03:35:57.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-10T03:35:57.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-10T03:35:57.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: legacy client fscid: -1 2024-09-10T03:35:57.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: 2024-09-10T03:35:57.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: No filesystems configured 2024-09-10T03:35:57.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mon.smithi169@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-10T03:35:57.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mon.smithi169@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-10T03:35:57.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mon.smithi169@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-10T03:35:57.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mon.smithi169@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-10T03:35:57.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mon.smithi169@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-10T03:35:57.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mon.smithi169@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-10T03:35:57.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mon.smithi169@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-09-10T03:35:57.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mon.smithi169@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-10T03:35:57.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mon.smithi169@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T03:35:57.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mon.smithi169@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T03:35:57.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mon.smithi169@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T03:35:57.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi169", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:35:57.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Deploying cephadm binary to smithi169 2024-09-10T03:35:57.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.982 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.982 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.982 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi112", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-10T03:35:57.982 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi112", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-09-10T03:35:57.982 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:35:57.982 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Deploying daemon crash.smithi112 on smithi112 2024-09-10T03:35:57.982 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.982 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Added host smithi169 2024-09-10T03:35:57.982 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.14189 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T03:35:57.982 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.982 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.982 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.982 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.983 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.983 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Deploying daemon node-exporter.smithi112 on smithi112 2024-09-10T03:35:57.983 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.983 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3550454350' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-10T03:35:57.983 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3550454350' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-10T03:35:57.983 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T03:35:57.983 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.983 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi112:172.21.15.112=smithi112;smithi169:172.21.15.169=smithi169", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:35:57.983 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Saving service mon spec with placement smithi112:172.21.15.112=smithi112;smithi169:172.21.15.169=smithi169;count:2 2024-09-10T03:35:57.983 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.983 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.983 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.983 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.984 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.984 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.984 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Deploying daemon alertmanager.smithi112 on smithi112 2024-09-10T03:35:57.984 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/423084877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.984 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.984 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.984 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/2140320670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.984 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.984 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/3103075283' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.984 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.984 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/1105217764' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.984 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.984 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.984 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.985 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.985 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.985 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.985 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.985 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-10T03:35:57.985 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.985 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.985 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T03:35:57.985 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.985 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T03:35:57.985 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Deploying daemon grafana.smithi112 on smithi112 2024-09-10T03:35:57.985 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.985 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/1051871552' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.986 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.986 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.986 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/1158040240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.986 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.986 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/540773876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.986 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.986 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.986 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.986 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.986 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.986 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.986 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.986 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.986 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.986 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.987 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Deploying daemon prometheus.smithi112 on smithi112 2024-09-10T03:35:57.987 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/944806810' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.987 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.987 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.987 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/3695935674' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.987 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.987 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/2692001257' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.987 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.987 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.987 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/2810367880' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.988 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.988 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/2354773801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.988 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.988 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.988 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.988 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.988 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-10T03:35:57.988 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/3753144255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.988 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.988 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14162 172.21.15.112:0/880103' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-10T03:35:57.988 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mgrmap e14: smithi112.pqrlrb(active, since 56s) 2024-09-10T03:35:57.988 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/3760111480' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.989 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/1859984990' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.989 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Active manager daemon smithi112.pqrlrb restarted 2024-09-10T03:35:57.989 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Activating manager daemon smithi112.pqrlrb 2024-09-10T03:35:57.989 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: osdmap e5: 0 total, 0 up, 0 in 2024-09-10T03:35:57.989 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mgrmap e15: smithi112.pqrlrb(active, starting, since 0.00330035s) 2024-09-10T03:35:57.989 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2024-09-10T03:35:57.989 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr metadata", "who": "smithi112.pqrlrb", "id": "smithi112.pqrlrb"}]: dispatch 2024-09-10T03:35:57.989 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T03:35:57.989 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T03:35:57.989 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T03:35:57.989 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Manager daemon smithi112.pqrlrb is now available 2024-09-10T03:35:57.989 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.989 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.990 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:35:57.990 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi112.pqrlrb/mirror_snapshot_schedule"}]: dispatch 2024-09-10T03:35:57.990 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi112.pqrlrb/trash_purge_schedule"}]: dispatch 2024-09-10T03:35:57.990 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.990 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mgrmap e16: smithi112.pqrlrb(active, since 1.00626s) 2024-09-10T03:35:57.990 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.990 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: [10/Sep/2024:03:35:36] ENGINE Bus STARTING 2024-09-10T03:35:57.990 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: [10/Sep/2024:03:35:36] ENGINE Serving on http://172.21.15.112:8765 2024-09-10T03:35:57.990 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: [10/Sep/2024:03:35:36] ENGINE Serving on https://172.21.15.112:7150 2024-09-10T03:35:57.990 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: [10/Sep/2024:03:35:36] ENGINE Bus STARTED 2024-09-10T03:35:57.990 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: [10/Sep/2024:03:35:36] ENGINE Error in HTTPServer.serve 2024-09-10T03:35:57.990 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Traceback (most recent call last): 2024-09-10T03:35:57.990 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T03:35:57.990 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: self._connections.run(self.expiration_interval) 2024-09-10T03:35:57.990 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T03:35:57.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: self._run(expiration_interval) 2024-09-10T03:35:57.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T03:35:57.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T03:35:57.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T03:35:57.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T03:35:57.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T03:35:57.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: s = self.context.wrap_socket( 2024-09-10T03:35:57.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T03:35:57.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: return self.sslsocket_class._create( 2024-09-10T03:35:57.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T03:35:57.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: self.do_handshake() 2024-09-10T03:35:57.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T03:35:57.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: self._sslobj.do_handshake() 2024-09-10T03:35:57.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T03:35:57.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/1572000195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.992 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.992 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mgrmap e17: smithi112.pqrlrb(active, since 2s) 2024-09-10T03:35:57.992 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.992 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.992 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/1615686960' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.992 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.992 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.992 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:35:57.992 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.992 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.992 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.992 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.992 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd/host:smithi169", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:35:57.992 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:35:57.993 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:35:57.993 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Updating smithi112:/etc/ceph/ceph.conf 2024-09-10T03:35:57.993 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Updating smithi169:/etc/ceph/ceph.conf 2024-09-10T03:35:57.993 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Updating smithi112:/var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T03:35:57.993 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Updating smithi169:/var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T03:35:57.993 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-09-10T03:35:57.993 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2024-09-10T03:35:57.993 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Updating smithi112:/var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T03:35:57.993 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Updating smithi169:/var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T03:35:57.993 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.993 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.993 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.993 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.994 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.994 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi169", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-10T03:35:57.994 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi169", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-09-10T03:35:57.994 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:35:57.994 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Deploying daemon ceph-exporter.smithi169 on smithi169 2024-09-10T03:35:57.994 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/3298124962' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.994 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.994 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.994 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.994 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.994 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi169", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-10T03:35:57.994 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi169", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-09-10T03:35:57.994 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:35:57.994 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Deploying daemon crash.smithi169 on smithi169 2024-09-10T03:35:57.995 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.995 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/1191909969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.995 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.995 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.995 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.995 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.995 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Deploying daemon node-exporter.smithi169 on smithi169 2024-09-10T03:35:57.995 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/1541531452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.995 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.995 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:35:57.995 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.995 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.995 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.995 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.996 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi169.bkwgjl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T03:35:57.996 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi169.bkwgjl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-10T03:35:57.996 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T03:35:57.996 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:35:57.996 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Deploying daemon mgr.smithi169.bkwgjl on smithi169 2024-09-10T03:35:57.996 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/3228768144' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:35:57.996 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.996 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.996 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.996 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.996 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T03:35:57.996 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:35:57.996 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: Deploying daemon mon.smithi169 on smithi169 2024-09-10T03:35:57.996 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:35:57.997 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:35:57.997 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 ceph-mon[35668]: mon.smithi169@-1(synchronizing).paxosservice(auth 1..8) refresh upgraded, format 0 -> 3 2024-09-10T03:35:57.997 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 bash[35637]: 38476c275eb5c2bb3e740552272aa1188a03e6e1d8775e55292d039a92bb9487 2024-09-10T03:35:57.997 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:35:57 smithi169 systemd[1]: Started Ceph mon.smithi169 for 513b3bc6-6f25-11ef-bcea-c7b262605968. 2024-09-10T03:35:59.316 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-10T03:35:59.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T03:35:59.560 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi169/config 2024-09-10T03:36:03.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2024-09-10T03:36:03.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: mon.smithi112 calling monitor election 2024-09-10T03:36:03.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:36:03.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:36:03.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:03.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: from='mgr.? 172.21.15.169:0/1512402971' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi169.bkwgjl/crt"}]: dispatch 2024-09-10T03:36:03.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:36:03.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: mon.smithi169 calling monitor election 2024-09-10T03:36:03.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:36:03.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:03.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:36:03.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:36:03.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: mon.smithi112 is new leader, mons smithi112,smithi169 in quorum (ranks 0,1) 2024-09-10T03:36:03.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: monmap epoch 2 2024-09-10T03:36:03.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:36:03.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: last_changed 2024-09-10T03:35:57.752257+0000 2024-09-10T03:36:03.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: created 2024-09-10T03:33:28.238100+0000 2024-09-10T03:36:03.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: min_mon_release 19 (squid) 2024-09-10T03:36:03.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: election_strategy: 1 2024-09-10T03:36:03.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: 0: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.smithi112 2024-09-10T03:36:03.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: 1: [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon.smithi169 2024-09-10T03:36:03.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: fsmap 2024-09-10T03:36:03.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: osdmap e5: 0 total, 0 up, 0 in 2024-09-10T03:36:03.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: mgrmap e17: smithi112.pqrlrb(active, since 27s) 2024-09-10T03:36:03.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: overall HEALTH_OK 2024-09-10T03:36:03.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: Standby manager daemon smithi169.bkwgjl started 2024-09-10T03:36:03.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:03.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: from='mgr.? 172.21.15.169:0/1512402971' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T03:36:03.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: from='mgr.? 172.21.15.169:0/1512402971' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi169.bkwgjl/key"}]: dispatch 2024-09-10T03:36:03.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:03.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: from='mgr.? 172.21.15.169:0/1512402971' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T03:36:03.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:03.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:03.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:02 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:03.269 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:36:03.270 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":2,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","modified":"2024-09-10T03:35:57.752257Z","created":"2024-09-10T03:33:28.238100Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi112","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-10T03:36:03.270 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 2 2024-09-10T03:36:03.270 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2024-09-10T03:36:03.270 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: mon.smithi112 calling monitor election 2024-09-10T03:36:03.270 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:36:03.270 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:36:03.271 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:03.271 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: from='mgr.? 172.21.15.169:0/1512402971' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi169.bkwgjl/crt"}]: dispatch 2024-09-10T03:36:03.271 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:36:03.271 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: mon.smithi169 calling monitor election 2024-09-10T03:36:03.271 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:36:03.271 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:03.271 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:36:03.271 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:36:03.271 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: mon.smithi112 is new leader, mons smithi112,smithi169 in quorum (ranks 0,1) 2024-09-10T03:36:03.271 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: monmap epoch 2 2024-09-10T03:36:03.271 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:36:03.271 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: last_changed 2024-09-10T03:35:57.752257+0000 2024-09-10T03:36:03.271 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: created 2024-09-10T03:33:28.238100+0000 2024-09-10T03:36:03.271 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: min_mon_release 19 (squid) 2024-09-10T03:36:03.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: election_strategy: 1 2024-09-10T03:36:03.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: 0: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.smithi112 2024-09-10T03:36:03.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: 1: [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon.smithi169 2024-09-10T03:36:03.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: fsmap 2024-09-10T03:36:03.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: osdmap e5: 0 total, 0 up, 0 in 2024-09-10T03:36:03.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: mgrmap e17: smithi112.pqrlrb(active, since 27s) 2024-09-10T03:36:03.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: overall HEALTH_OK 2024-09-10T03:36:03.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: Standby manager daemon smithi169.bkwgjl started 2024-09-10T03:36:03.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:03.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: from='mgr.? 172.21.15.169:0/1512402971' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T03:36:03.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: from='mgr.? 172.21.15.169:0/1512402971' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi169.bkwgjl/key"}]: dispatch 2024-09-10T03:36:03.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:03.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: from='mgr.? 172.21.15.169:0/1512402971' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T03:36:03.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:03.273 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:03.273 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:02 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:03.911 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:03 smithi169 ceph-mon[35668]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:03.911 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:03 smithi169 ceph-mon[35668]: mgrmap e18: smithi112.pqrlrb(active, since 28s), standbys: smithi169.bkwgjl 2024-09-10T03:36:03.911 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:03 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr metadata", "who": "smithi169.bkwgjl", "id": "smithi169.bkwgjl"}]: dispatch 2024-09-10T03:36:03.911 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:03 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:36:03.911 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:03 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/1509231398' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:36:03.911 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:03 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:36:04.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:03 smithi112 ceph-mon[27737]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:04.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:03 smithi112 ceph-mon[27737]: mgrmap e18: smithi112.pqrlrb(active, since 28s), standbys: smithi169.bkwgjl 2024-09-10T03:36:04.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:03 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr metadata", "who": "smithi169.bkwgjl", "id": "smithi169.bkwgjl"}]: dispatch 2024-09-10T03:36:04.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:03 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:36:04.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:03 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/1509231398' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T03:36:04.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:03 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:36:04.541 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-10T03:36:04.542 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config generate-minimal-conf 2024-09-10T03:36:04.801 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:05.557 INFO:teuthology.orchestra.run.smithi112.stdout:# minimal ceph.conf for 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:36:05.557 INFO:teuthology.orchestra.run.smithi112.stdout:[global] 2024-09-10T03:36:05.557 INFO:teuthology.orchestra.run.smithi112.stdout: fsid = 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T03:36:05.557 INFO:teuthology.orchestra.run.smithi112.stdout: mon_host = [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] 2024-09-10T03:36:06.166 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:05 smithi112 ceph-mon[27737]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:06.166 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:36:06.166 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:06.166 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:06.166 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:06.166 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:06.166 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:36:06.166 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:05 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2941332191' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:06.167 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:06.167 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:06.167 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:06.167 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:06.167 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:06.167 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T03:36:06.167 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T03:36:06.167 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:06.193 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-10T03:36:06.194 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T03:36:06.194 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.conf 2024-09-10T03:36:06.222 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T03:36:06.222 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-10T03:36:06.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:05 smithi169 ceph-mon[35668]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:06.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:36:06.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:06.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:06.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:06.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:06.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:36:06.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:05 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/2941332191' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:06.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:06.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:06.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:06.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:06.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:06.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T03:36:06.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T03:36:06.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:06.288 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-10T03:36:06.288 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.conf 2024-09-10T03:36:06.316 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-10T03:36:06.317 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-10T03:36:06.382 INFO:tasks.cephadm:setting config [client] client_mount_timeout = 600 2024-09-10T03:36:06.382 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set client client_mount_timeout 600 2024-09-10T03:36:06.384 INFO:tasks.cephadm:setting config [client] debug_client = 20 2024-09-10T03:36:06.385 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set client debug_client 20 2024-09-10T03:36:06.428 INFO:tasks.cephadm:setting config [client] debug_ms = 1 2024-09-10T03:36:06.428 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set client debug_ms 1 2024-09-10T03:36:06.471 INFO:tasks.cephadm:setting config [client] rados_mon_op_timeout = 900 2024-09-10T03:36:06.472 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set client rados_mon_op_timeout 900 2024-09-10T03:36:06.515 INFO:tasks.cephadm:setting config [client] rados_osd_op_timeout = 900 2024-09-10T03:36:06.515 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set client rados_osd_op_timeout 900 2024-09-10T03:36:06.561 INFO:tasks.cephadm:setting config [mds] debug_mds = 20 2024-09-10T03:36:06.562 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set mds debug_mds 20 2024-09-10T03:36:06.605 INFO:tasks.cephadm:setting config [mds] debug_mds_balancer = 20 2024-09-10T03:36:06.605 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set mds debug_mds_balancer 20 2024-09-10T03:36:06.648 INFO:tasks.cephadm:setting config [mds] debug_ms = 2 2024-09-10T03:36:06.648 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set mds debug_ms 2 2024-09-10T03:36:06.691 INFO:tasks.cephadm:setting config [mds] mds_debug_frag = True 2024-09-10T03:36:06.692 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set mds mds_debug_frag True 2024-09-10T03:36:06.740 INFO:tasks.cephadm:setting config [mds] mds_debug_scatterstat = True 2024-09-10T03:36:06.759 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set mds mds_debug_scatterstat True 2024-09-10T03:36:06.803 INFO:tasks.cephadm:setting config [mds] mds_op_complaint_time = 180 2024-09-10T03:36:06.803 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set mds mds_op_complaint_time 180 2024-09-10T03:36:06.847 INFO:tasks.cephadm:setting config [mds] mds_verify_scatter = True 2024-09-10T03:36:06.847 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set mds mds_verify_scatter True 2024-09-10T03:36:06.890 INFO:tasks.cephadm:setting config [mds] osd_op_complaint_time = 180 2024-09-10T03:36:06.890 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set mds osd_op_complaint_time 180 2024-09-10T03:36:06.934 INFO:tasks.cephadm:setting config [mds] rados_mon_op_timeout = 900 2024-09-10T03:36:06.934 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set mds rados_mon_op_timeout 900 2024-09-10T03:36:06.959 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:06 smithi112 ceph-mon[27737]: Updating smithi112:/etc/ceph/ceph.conf 2024-09-10T03:36:06.960 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:06 smithi112 ceph-mon[27737]: Updating smithi169:/etc/ceph/ceph.conf 2024-09-10T03:36:06.960 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:06 smithi112 ceph-mon[27737]: Updating smithi169:/var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T03:36:06.960 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:06 smithi112 ceph-mon[27737]: Updating smithi112:/var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T03:36:06.960 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:06 smithi112 ceph-mon[27737]: Reconfiguring mon.smithi112 (unknown last config time)... 2024-09-10T03:36:06.960 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:06 smithi112 ceph-mon[27737]: Reconfiguring daemon mon.smithi112 on smithi112 2024-09-10T03:36:06.961 INFO:tasks.cephadm:setting config [mds] rados_osd_op_timeout = 900 2024-09-10T03:36:06.961 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set mds rados_osd_op_timeout 900 2024-09-10T03:36:07.004 INFO:tasks.cephadm:setting config [mon] mon_op_complaint_time = 120 2024-09-10T03:36:07.004 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set mon mon_op_complaint_time 120 2024-09-10T03:36:07.047 INFO:tasks.cephadm:setting config [mon] mon_warn_on_pool_no_app = False 2024-09-10T03:36:07.047 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set mon mon_warn_on_pool_no_app False 2024-09-10T03:36:07.093 INFO:tasks.cephadm:setting config [osd] osd_op_complaint_time = 180 2024-09-10T03:36:07.093 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config set osd osd_op_complaint_time 180 2024-09-10T03:36:07.149 DEBUG:tasks.cephadm:set 18 configs 2024-09-10T03:36:07.149 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi112') 2024-09-10T03:36:07.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:06 smithi169 ceph-mon[35668]: Updating smithi112:/etc/ceph/ceph.conf 2024-09-10T03:36:07.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:06 smithi169 ceph-mon[35668]: Updating smithi169:/etc/ceph/ceph.conf 2024-09-10T03:36:07.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:06 smithi169 ceph-mon[35668]: Updating smithi169:/var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T03:36:07.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:06 smithi169 ceph-mon[35668]: Updating smithi112:/var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T03:36:07.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:06 smithi169 ceph-mon[35668]: Reconfiguring mon.smithi112 (unknown last config time)... 2024-09-10T03:36:07.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:06 smithi169 ceph-mon[35668]: Reconfiguring daemon mon.smithi112 on smithi112 2024-09-10T03:36:07.316 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:07.323 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:07.323 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:07.323 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:07.324 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:08.120 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:07 smithi112 ceph-mon[27737]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:08.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:07 smithi169 ceph-mon[35668]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:09.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:09.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:09.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:09 smithi169 ceph-mon[35668]: Reconfiguring mgr.smithi112.pqrlrb (unknown last config time)... 2024-09-10T03:36:09.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi112.pqrlrb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T03:36:09.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T03:36:09.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:09.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:09 smithi169 ceph-mon[35668]: Reconfiguring daemon mgr.smithi112.pqrlrb on smithi112 2024-09-10T03:36:09.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:09 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/1322040679' entity='client.admin' 2024-09-10T03:36:09.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:09.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:09.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:09 smithi112 ceph-mon[27737]: Reconfiguring mgr.smithi112.pqrlrb (unknown last config time)... 2024-09-10T03:36:09.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi112.pqrlrb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T03:36:09.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T03:36:09.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:09.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:09 smithi112 ceph-mon[27737]: Reconfiguring daemon mgr.smithi112.pqrlrb on smithi112 2024-09-10T03:36:09.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:09 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/1322040679' entity='client.admin' 2024-09-10T03:36:10.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:10 smithi112 ceph-mon[27737]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:10.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:10 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/424894101' entity='client.admin' 2024-09-10T03:36:10.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:10 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/665166732' entity='client.admin' 2024-09-10T03:36:10.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:10 smithi169 ceph-mon[35668]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:10.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:10 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/424894101' entity='client.admin' 2024-09-10T03:36:10.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:10 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/665166732' entity='client.admin' 2024-09-10T03:36:12.239 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:12 smithi112 ceph-mon[27737]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:12.240 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:12 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/524925744' entity='client.admin' 2024-09-10T03:36:12.328 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:12 smithi169 ceph-mon[35668]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:12.328 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:12 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/524925744' entity='client.admin' 2024-09-10T03:36:12.783 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:13.429 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:13.430 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:13.430 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:13.433 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:13.433 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:13.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:13 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2202557295' entity='client.admin' 2024-09-10T03:36:13.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:13 smithi112 ceph-mon[27737]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:13.970 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:13 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/2202557295' entity='client.admin' 2024-09-10T03:36:13.971 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:13 smithi169 ceph-mon[35668]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:14.644 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:14.644 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:15.243 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:14 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:15.244 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:14 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:15.244 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:14 smithi112 ceph-mon[27737]: Reconfiguring ceph-exporter.smithi112 (monmap changed)... 2024-09-10T03:36:15.244 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:14 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi112", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-10T03:36:15.244 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:14 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:15.244 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:14 smithi112 ceph-mon[27737]: Reconfiguring daemon ceph-exporter.smithi112 on smithi112 2024-09-10T03:36:15.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:14 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:15.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:14 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:15.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:14 smithi169 ceph-mon[35668]: Reconfiguring ceph-exporter.smithi112 (monmap changed)... 2024-09-10T03:36:15.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:14 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi112", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-10T03:36:15.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:14 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:15.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:14 smithi169 ceph-mon[35668]: Reconfiguring daemon ceph-exporter.smithi112 on smithi112 2024-09-10T03:36:15.808 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:15.808 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:16.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:15 smithi169 ceph-mon[35668]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:16.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:15 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/1464240211' entity='client.admin' 2024-09-10T03:36:16.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:15 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/2021011220' entity='client.admin' 2024-09-10T03:36:16.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:15 smithi112 ceph-mon[27737]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:16.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:15 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/1464240211' entity='client.admin' 2024-09-10T03:36:16.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:15 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2021011220' entity='client.admin' 2024-09-10T03:36:17.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:16 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/1970522312' entity='client.admin' 2024-09-10T03:36:17.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:16 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/1134082788' entity='client.admin' 2024-09-10T03:36:17.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:16 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/1970522312' entity='client.admin' 2024-09-10T03:36:17.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:16 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/1134082788' entity='client.admin' 2024-09-10T03:36:17.583 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:18.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:18 smithi112 ceph-mon[27737]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:18.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:18 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2502466287' entity='client.admin' 2024-09-10T03:36:18.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:18 smithi169 ceph-mon[35668]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:18.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:18 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/2502466287' entity='client.admin' 2024-09-10T03:36:20.222 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:20 smithi112 ceph-mon[27737]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:20.222 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:20 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3848739911' entity='client.admin' 2024-09-10T03:36:20.222 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:20 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:36:20.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:20 smithi169 ceph-mon[35668]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:20.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:20 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3848739911' entity='client.admin' 2024-09-10T03:36:20.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:20 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:36:20.669 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi112') 2024-09-10T03:36:20.669 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi112') 2024-09-10T03:36:21.773 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:21 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2064586369' entity='client.admin' 2024-09-10T03:36:21.773 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:21 smithi112 ceph-mon[27737]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:21.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:21 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/2064586369' entity='client.admin' 2024-09-10T03:36:21.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:21 smithi169 ceph-mon[35668]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:22.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:22.398+0000 7f18c4d98640 1 -- 172.21.15.112:0/504891277 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f18ac06ddb0 msgr2=0x7f18ac070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:22.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:22.398+0000 7f18c4d98640 1 --2- 172.21.15.112:0/504891277 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f18ac06ddb0 0x7f18ac070270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f18b0000c00 tx=0x7f18b0004eb0 comp rx=0 tx=0).stop 2024-09-10T03:36:22.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:22.398+0000 7f18c4d98640 1 -- 172.21.15.112:0/504891277 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18c0101a70 msgr2=0x7f18c019c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:22.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:22.398+0000 7f18c4d98640 1 --2- 172.21.15.112:0/504891277 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18c0101a70 0x7f18c019c120 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f18a800c8d0 tx=0x7f18a800cd90 comp rx=0 tx=0).stop 2024-09-10T03:36:22.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:22.398+0000 7f18c4d98640 1 -- 172.21.15.112:0/504891277 shutdown_connections 2024-09-10T03:36:22.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:22.398+0000 7f18c4d98640 1 --2- 172.21.15.112:0/504891277 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f18ac06ddb0 0x7f18ac070270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:22.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:22.398+0000 7f18c4d98640 1 --2- 172.21.15.112:0/504891277 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18c0101a70 0x7f18c019c120 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:22.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:22.398+0000 7f18c4d98640 1 --2- 172.21.15.112:0/504891277 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f18c0100870 0x7f18c006ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:22.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:22.398+0000 7f18c4d98640 1 -- 172.21.15.112:0/504891277 >> 172.21.15.112:0/504891277 conn(0x7f18c00fc020 msgr2=0x7f18c00fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:36:22.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:22.398+0000 7f18c4d98640 1 -- 172.21.15.112:0/504891277 shutdown_connections 2024-09-10T03:36:22.401 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:22.398+0000 7f18c4d98640 1 -- 172.21.15.112:0/504891277 wait complete. 2024-09-10T03:36:23.338 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi112') 2024-09-10T03:36:23.338 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi112') 2024-09-10T03:36:23.338 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi112') 2024-09-10T03:36:23.435 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:23 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/4171169556' entity='client.admin' 2024-09-10T03:36:23.436 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:23 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/504891277' entity='client.admin' 2024-09-10T03:36:23.436 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:23 smithi112 ceph-mon[27737]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:23.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.517+0000 7f20be96c640 1 -- 172.21.15.112:0/3147693012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20b8100880 msgr2=0x7f20b8100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:23.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.517+0000 7f20be96c640 1 --2- 172.21.15.112:0/3147693012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20b8100880 0x7f20b8100c80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f20a8009880 tx=0x7f20a802a4a0 comp rx=0 tx=0).stop 2024-09-10T03:36:23.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.517+0000 7f20be96c640 1 -- 172.21.15.112:0/3147693012 shutdown_connections 2024-09-10T03:36:23.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.517+0000 7f20be96c640 1 --2- 172.21.15.112:0/3147693012 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20b8101a80 0x7f20b8101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:23.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.517+0000 7f20be96c640 1 --2- 172.21.15.112:0/3147693012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20b8100880 0x7f20b8100c80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:23.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.517+0000 7f20be96c640 1 -- 172.21.15.112:0/3147693012 >> 172.21.15.112:0/3147693012 conn(0x7f20b80fc010 msgr2=0x7f20b80fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:36:23.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.518+0000 7f20be96c640 1 -- 172.21.15.112:0/3147693012 shutdown_connections 2024-09-10T03:36:23.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.518+0000 7f20be96c640 1 -- 172.21.15.112:0/3147693012 wait complete. 2024-09-10T03:36:23.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.518+0000 7f20be96c640 1 Processor -- start 2024-09-10T03:36:23.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.518+0000 7f20be96c640 1 -- start start 2024-09-10T03:36:23.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.519+0000 7f20be96c640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20b8100880 0x7f20b806adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:36:23.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.519+0000 7f20be96c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20b8101a80 0x7f20b819c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:36:23.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.519+0000 7f20be96c640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f20b81096a0 con 0x7f20b8101a80 2024-09-10T03:36:23.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.519+0000 7f20be96c640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f20b8109520 con 0x7f20b8100880 2024-09-10T03:36:23.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.519+0000 7f20affff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20b8101a80 0x7f20b819c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:36:23.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.519+0000 7f20affff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20b8101a80 0x7f20b819c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:57564/0 (socket says 172.21.15.112:57564) 2024-09-10T03:36:23.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.519+0000 7f20affff640 1 -- 172.21.15.112:0/2655360581 learned_addr learned my addr 172.21.15.112:0/2655360581 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:36:23.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.519+0000 7f20b7fff640 1 --2- 172.21.15.112:0/2655360581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20b8100880 0x7f20b806adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:36:23.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.519+0000 7f20affff640 1 -- 172.21.15.112:0/2655360581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20b8100880 msgr2=0x7f20b806adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:23.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.519+0000 7f20affff640 1 --2- 172.21.15.112:0/2655360581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20b8100880 0x7f20b806adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:23.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.519+0000 7f20affff640 1 -- 172.21.15.112:0/2655360581 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b819c730 con 0x7f20b8101a80 2024-09-10T03:36:23.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.519+0000 7f20affff640 1 --2- 172.21.15.112:0/2655360581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20b8101a80 0x7f20b819c1f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f20a000c8d0 tx=0x7f20a000cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:36:23.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.519+0000 7f20b5ffb640 1 -- 172.21.15.112:0/2655360581 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f20a00079b0 con 0x7f20b8101a80 2024-09-10T03:36:23.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.519+0000 7f20be96c640 1 -- 172.21.15.112:0/2655360581 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20b819ca20 con 0x7f20b8101a80 2024-09-10T03:36:23.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.519+0000 7f20b5ffb640 1 -- 172.21.15.112:0/2655360581 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f20a00054e0 con 0x7f20b8101a80 2024-09-10T03:36:23.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.519+0000 7f20b5ffb640 1 -- 172.21.15.112:0/2655360581 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f20a0004e40 con 0x7f20b8101a80 2024-09-10T03:36:23.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.519+0000 7f20be96c640 1 -- 172.21.15.112:0/2655360581 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20b819cfb0 con 0x7f20b8101a80 2024-09-10T03:36:23.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.521+0000 7f20be96c640 1 -- 172.21.15.112:0/2655360581 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20b8100c80 con 0x7f20b8101a80 2024-09-10T03:36:23.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.521+0000 7f20b5ffb640 1 -- 172.21.15.112:0/2655360581 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 18) ==== 92149+0+0 (secure 0 0 0) 0x7f20a0007b50 con 0x7f20b8101a80 2024-09-10T03:36:23.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.521+0000 7f20b5ffb640 1 --2- 172.21.15.112:0/2655360581 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f20a406ddb0 0x7f20a4070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:36:23.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.521+0000 7f20b5ffb640 1 -- 172.21.15.112:0/2655360581 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f20a008ca20 con 0x7f20b8101a80 2024-09-10T03:36:23.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.521+0000 7f20b7fff640 1 --2- 172.21.15.112:0/2655360581 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f20a406ddb0 0x7f20a4070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:36:23.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.522+0000 7f20b7fff640 1 --2- 172.21.15.112:0/2655360581 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f20a406ddb0 0x7f20a4070270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f20a8009230 tx=0x7f20a80058d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:36:23.525 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:23.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:23 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/4171169556' entity='client.admin' 2024-09-10T03:36:23.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:23 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/504891277' entity='client.admin' 2024-09-10T03:36:23.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:23 smithi169 ceph-mon[35668]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:23.526 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.526+0000 7f20b5ffb640 1 -- 172.21.15.112:0/2655360581 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f20a0014030 con 0x7f20b8101a80 2024-09-10T03:36:23.699 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.698+0000 7f20be96c640 1 -- 172.21.15.112:0/2655360581 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f20b819d210 con 0x7f20b8101a80 2024-09-10T03:36:23.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.702+0000 7f20b5ffb640 1 -- 172.21.15.112:0/2655360581 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v25)=0 v25) ==== 120+0+0 (secure 0 0 0) 0x7f20a005f350 con 0x7f20b8101a80 2024-09-10T03:36:23.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.704+0000 7f20be96c640 1 -- 172.21.15.112:0/2655360581 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f20a406ddb0 msgr2=0x7f20a4070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:23.705 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.704+0000 7f20be96c640 1 --2- 172.21.15.112:0/2655360581 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f20a406ddb0 0x7f20a4070270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f20a8009230 tx=0x7f20a80058d0 comp rx=0 tx=0).stop 2024-09-10T03:36:23.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.704+0000 7f20be96c640 1 -- 172.21.15.112:0/2655360581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20b8101a80 msgr2=0x7f20b819c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:23.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.704+0000 7f20be96c640 1 --2- 172.21.15.112:0/2655360581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20b8101a80 0x7f20b819c1f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f20a000c8d0 tx=0x7f20a000cd90 comp rx=0 tx=0).stop 2024-09-10T03:36:23.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.704+0000 7f20be96c640 1 -- 172.21.15.112:0/2655360581 shutdown_connections 2024-09-10T03:36:23.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.705+0000 7f20be96c640 1 --2- 172.21.15.112:0/2655360581 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f20a406ddb0 0x7f20a4070270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:23.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.705+0000 7f20be96c640 1 --2- 172.21.15.112:0/2655360581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20b8101a80 0x7f20b819c1f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:23.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.705+0000 7f20be96c640 1 --2- 172.21.15.112:0/2655360581 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f20b8100880 0x7f20b806adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:23.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.705+0000 7f20be96c640 1 -- 172.21.15.112:0/2655360581 >> 172.21.15.112:0/2655360581 conn(0x7f20b80fc010 msgr2=0x7f20b80fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:36:23.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.705+0000 7f20be96c640 1 -- 172.21.15.112:0/2655360581 shutdown_connections 2024-09-10T03:36:23.706 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:23.705+0000 7f20be96c640 1 -- 172.21.15.112:0/2655360581 wait complete. 2024-09-10T03:36:24.512 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:24.854 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:24 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2655360581' entity='client.admin' 2024-09-10T03:36:24.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.934+0000 7fef774ca640 1 -- 172.21.15.112:0/4136335715 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70100880 msgr2=0x7fef70100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:24.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.934+0000 7fef774ca640 1 --2- 172.21.15.112:0/4136335715 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70100880 0x7fef70100c80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fef6c009880 tx=0x7fef6c02a4a0 comp rx=0 tx=0).stop 2024-09-10T03:36:24.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.934+0000 7fef774ca640 1 -- 172.21.15.112:0/4136335715 shutdown_connections 2024-09-10T03:36:24.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.934+0000 7fef774ca640 1 --2- 172.21.15.112:0/4136335715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef70101a80 0x7fef70101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:24.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.934+0000 7fef774ca640 1 --2- 172.21.15.112:0/4136335715 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70100880 0x7fef70100c80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:24.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.934+0000 7fef774ca640 1 -- 172.21.15.112:0/4136335715 >> 172.21.15.112:0/4136335715 conn(0x7fef700fc010 msgr2=0x7fef700fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:36:24.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.934+0000 7fef774ca640 1 -- 172.21.15.112:0/4136335715 shutdown_connections 2024-09-10T03:36:24.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.934+0000 7fef774ca640 1 -- 172.21.15.112:0/4136335715 wait complete. 2024-09-10T03:36:24.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.934+0000 7fef774ca640 1 Processor -- start 2024-09-10T03:36:24.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.934+0000 7fef774ca640 1 -- start start 2024-09-10T03:36:24.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.935+0000 7fef774ca640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef70100880 0x7fef70196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:36:24.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.935+0000 7fef774ca640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70101a80 0x7fef70196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:36:24.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.935+0000 7fef774ca640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fef70109720 con 0x7fef70101a80 2024-09-10T03:36:24.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.935+0000 7fef774ca640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fef701095a0 con 0x7fef70100880 2024-09-10T03:36:24.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.935+0000 7fef74a65640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70101a80 0x7fef70196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:36:24.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.935+0000 7fef74a65640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70101a80 0x7fef70196950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:57580/0 (socket says 172.21.15.112:57580) 2024-09-10T03:36:24.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.935+0000 7fef74a65640 1 -- 172.21.15.112:0/1324947697 learned_addr learned my addr 172.21.15.112:0/1324947697 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:36:24.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.935+0000 7fef74a65640 1 -- 172.21.15.112:0/1324947697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef70100880 msgr2=0x7fef70196410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:24.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.935+0000 7fef75266640 1 --2- 172.21.15.112:0/1324947697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef70100880 0x7fef70196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:36:24.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.935+0000 7fef74a65640 1 --2- 172.21.15.112:0/1324947697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef70100880 0x7fef70196410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:24.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.935+0000 7fef74a65640 1 -- 172.21.15.112:0/1324947697 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef70196fd0 con 0x7fef70101a80 2024-09-10T03:36:24.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.935+0000 7fef74a65640 1 --2- 172.21.15.112:0/1324947697 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70101a80 0x7fef70196950 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fef6000d910 tx=0x7fef6000ddd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:36:24.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.935+0000 7fef667fc640 1 -- 172.21.15.112:0/1324947697 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fef6001b070 con 0x7fef70101a80 2024-09-10T03:36:24.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.935+0000 7fef667fc640 1 -- 172.21.15.112:0/1324947697 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fef6000eb70 con 0x7fef70101a80 2024-09-10T03:36:24.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.935+0000 7fef774ca640 1 -- 172.21.15.112:0/1324947697 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef7006a360 con 0x7fef70101a80 2024-09-10T03:36:24.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.935+0000 7fef667fc640 1 -- 172.21.15.112:0/1324947697 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fef60014600 con 0x7fef70101a80 2024-09-10T03:36:24.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.935+0000 7fef774ca640 1 -- 172.21.15.112:0/1324947697 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef7006a970 con 0x7fef70101a80 2024-09-10T03:36:24.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.937+0000 7fef774ca640 1 -- 172.21.15.112:0/1324947697 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef70100c80 con 0x7fef70101a80 2024-09-10T03:36:24.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.937+0000 7fef667fc640 1 -- 172.21.15.112:0/1324947697 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 18) ==== 92149+0+0 (secure 0 0 0) 0x7fef6001e070 con 0x7fef70101a80 2024-09-10T03:36:24.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.937+0000 7fef667fc640 1 --2- 172.21.15.112:0/1324947697 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fef5c06de00 0x7fef5c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:36:24.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.938+0000 7fef667fc640 1 -- 172.21.15.112:0/1324947697 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fef6008e360 con 0x7fef70101a80 2024-09-10T03:36:24.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.938+0000 7fef75266640 1 --2- 172.21.15.112:0/1324947697 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fef5c06de00 0x7fef5c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:36:24.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.938+0000 7fef75266640 1 --2- 172.21.15.112:0/1324947697 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fef5c06de00 0x7fef5c0702c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fef6c006010 tx=0x7fef6c000bc0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:36:24.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:24.942+0000 7fef667fc640 1 -- 172.21.15.112:0/1324947697 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fef6005d060 con 0x7fef70101a80 2024-09-10T03:36:25.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:24 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/2655360581' entity='client.admin' 2024-09-10T03:36:25.114 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:25.114+0000 7fef774ca640 1 -- 172.21.15.112:0/1324947697 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7fef7019b950 con 0x7fef70101a80 2024-09-10T03:36:25.118 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:25.117+0000 7fef667fc640 1 -- 172.21.15.112:0/1324947697 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v26)=0 v26) ==== 121+0+0 (secure 0 0 0) 0x7fef60060d10 con 0x7fef70101a80 2024-09-10T03:36:25.120 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:25.119+0000 7fef774ca640 1 -- 172.21.15.112:0/1324947697 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fef5c06de00 msgr2=0x7fef5c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:25.120 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:25.119+0000 7fef774ca640 1 --2- 172.21.15.112:0/1324947697 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fef5c06de00 0x7fef5c0702c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fef6c006010 tx=0x7fef6c000bc0 comp rx=0 tx=0).stop 2024-09-10T03:36:25.120 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:25.119+0000 7fef774ca640 1 -- 172.21.15.112:0/1324947697 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70101a80 msgr2=0x7fef70196950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:25.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:25.119+0000 7fef774ca640 1 --2- 172.21.15.112:0/1324947697 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70101a80 0x7fef70196950 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fef6000d910 tx=0x7fef6000ddd0 comp rx=0 tx=0).stop 2024-09-10T03:36:25.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:25.119+0000 7fef774ca640 1 -- 172.21.15.112:0/1324947697 shutdown_connections 2024-09-10T03:36:25.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:25.119+0000 7fef774ca640 1 --2- 172.21.15.112:0/1324947697 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fef5c06de00 0x7fef5c0702c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:25.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:25.119+0000 7fef774ca640 1 --2- 172.21.15.112:0/1324947697 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef70101a80 0x7fef70196950 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:25.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:25.119+0000 7fef774ca640 1 --2- 172.21.15.112:0/1324947697 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fef70100880 0x7fef70196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:25.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:25.119+0000 7fef774ca640 1 -- 172.21.15.112:0/1324947697 >> 172.21.15.112:0/1324947697 conn(0x7fef700fc010 msgr2=0x7fef700fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:36:25.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:25.119+0000 7fef774ca640 1 -- 172.21.15.112:0/1324947697 shutdown_connections 2024-09-10T03:36:25.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:25.119+0000 7fef774ca640 1 -- 172.21.15.112:0/1324947697 wait complete. 2024-09-10T03:36:26.290 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:26 smithi112 ceph-mon[27737]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:26.290 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:26 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/1324947697' entity='client.admin' 2024-09-10T03:36:26.290 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:26 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:26.290 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:26 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:26.290 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:26 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi112", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-10T03:36:26.290 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:26 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:26.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.480+0000 7f2fc1989640 1 -- 172.21.15.112:0/3033401995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2fbc101a60 msgr2=0x7f2fbc101ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:26.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.480+0000 7f2fc1989640 1 --2- 172.21.15.112:0/3033401995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2fbc101a60 0x7f2fbc101ee0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0009880 tx=0x7f2fb002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:36:26.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.480+0000 7f2fc1989640 1 -- 172.21.15.112:0/3033401995 shutdown_connections 2024-09-10T03:36:26.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.480+0000 7f2fc1989640 1 --2- 172.21.15.112:0/3033401995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2fbc101a60 0x7f2fbc101ee0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:26.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.480+0000 7f2fc1989640 1 --2- 172.21.15.112:0/3033401995 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fbc100860 0x7f2fbc100c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:26.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.480+0000 7f2fc1989640 1 -- 172.21.15.112:0/3033401995 >> 172.21.15.112:0/3033401995 conn(0x7f2fbc0fbff0 msgr2=0x7f2fbc0fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:36:26.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.480+0000 7f2fc1989640 1 -- 172.21.15.112:0/3033401995 shutdown_connections 2024-09-10T03:36:26.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.480+0000 7f2fc1989640 1 -- 172.21.15.112:0/3033401995 wait complete. 2024-09-10T03:36:26.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.480+0000 7f2fc1989640 1 Processor -- start 2024-09-10T03:36:26.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.480+0000 7f2fc1989640 1 -- start start 2024-09-10T03:36:26.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.481+0000 7f2fc1989640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2fbc100860 0x7f2fbc104780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:36:26.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.481+0000 7f2fc1989640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fbc101a60 0x7f2fbc102e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:36:26.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.481+0000 7f2fc1989640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc109680 con 0x7f2fbc100860 2024-09-10T03:36:26.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.481+0000 7f2fc1989640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc109500 con 0x7f2fbc101a60 2024-09-10T03:36:26.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.481+0000 7f2fba7fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fbc101a60 0x7f2fbc102e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:36:26.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.481+0000 7f2fba7fc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fbc101a60 0x7f2fbc102e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.112:36738/0 (socket says 172.21.15.112:36738) 2024-09-10T03:36:26.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.481+0000 7f2fba7fc640 1 -- 172.21.15.112:0/1099974635 learned_addr learned my addr 172.21.15.112:0/1099974635 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:36:26.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.481+0000 7f2fbaffd640 1 --2- 172.21.15.112:0/1099974635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2fbc100860 0x7f2fbc104780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:36:26.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.481+0000 7f2fbaffd640 1 -- 172.21.15.112:0/1099974635 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fbc101a60 msgr2=0x7f2fbc102e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:26.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.481+0000 7f2fbaffd640 1 --2- 172.21.15.112:0/1099974635 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fbc101a60 0x7f2fbc102e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:26.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.481+0000 7f2fbaffd640 1 -- 172.21.15.112:0/1099974635 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fbc103380 con 0x7f2fbc100860 2024-09-10T03:36:26.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.481+0000 7f2fba7fc640 1 --2- 172.21.15.112:0/1099974635 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fbc101a60 0x7f2fbc102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T03:36:26.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.481+0000 7f2fbaffd640 1 --2- 172.21.15.112:0/1099974635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2fbc100860 0x7f2fbc104780 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2fac00c8c0 tx=0x7f2fac00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:36:26.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.481+0000 7f2fc0987640 1 -- 172.21.15.112:0/1099974635 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2fac0079b0 con 0x7f2fbc100860 2024-09-10T03:36:26.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.481+0000 7f2fc1989640 1 -- 172.21.15.112:0/1099974635 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fbc103670 con 0x7f2fbc100860 2024-09-10T03:36:26.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.481+0000 7f2fc0987640 1 -- 172.21.15.112:0/1099974635 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2fac007b50 con 0x7f2fbc100860 2024-09-10T03:36:26.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.481+0000 7f2fc0987640 1 -- 172.21.15.112:0/1099974635 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2fac018630 con 0x7f2fbc100860 2024-09-10T03:36:26.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.482+0000 7f2fc1989640 1 -- 172.21.15.112:0/1099974635 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fbc103c00 con 0x7f2fbc100860 2024-09-10T03:36:26.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.483+0000 7f2fc1989640 1 -- 172.21.15.112:0/1099974635 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fbc100c60 con 0x7f2fbc100860 2024-09-10T03:36:26.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.483+0000 7f2fc0987640 1 -- 172.21.15.112:0/1099974635 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 18) ==== 92149+0+0 (secure 0 0 0) 0x7f2fac00f450 con 0x7f2fbc100860 2024-09-10T03:36:26.485 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.483+0000 7f2fc0987640 1 --2- 172.21.15.112:0/1099974635 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2fa406ddb0 0x7f2fa4070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:36:26.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.484+0000 7f2fc0987640 1 -- 172.21.15.112:0/1099974635 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f2fac08cbf0 con 0x7f2fbc100860 2024-09-10T03:36:26.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.484+0000 7f2fba7fc640 1 --2- 172.21.15.112:0/1099974635 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2fa406ddb0 0x7f2fa4070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:36:26.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.484+0000 7f2fba7fc640 1 --2- 172.21.15.112:0/1099974635 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2fa406ddb0 0x7f2fa4070270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2fb00096c0 tx=0x7f2fb0005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:36:26.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.488+0000 7f2fc0987640 1 -- 172.21.15.112:0/1099974635 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2fac014030 con 0x7f2fbc100860 2024-09-10T03:36:26.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:26 smithi169 ceph-mon[35668]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:26.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:26 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/1324947697' entity='client.admin' 2024-09-10T03:36:26.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:26 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:26.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:26 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:26.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:26 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi112", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-10T03:36:26.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:26 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:26.659 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.658+0000 7f2fc1989640 1 -- 172.21.15.112:0/1099974635 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f2fbc19f590 con 0x7f2fbc100860 2024-09-10T03:36:26.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.662+0000 7f2fc0987640 1 -- 172.21.15.112:0/1099974635 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v27)=0 v27) ==== 108+0+0 (secure 0 0 0) 0x7f2fac05b870 con 0x7f2fbc100860 2024-09-10T03:36:26.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.664+0000 7f2fc1989640 1 -- 172.21.15.112:0/1099974635 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2fa406ddb0 msgr2=0x7f2fa4070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:26.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.664+0000 7f2fc1989640 1 --2- 172.21.15.112:0/1099974635 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2fa406ddb0 0x7f2fa4070270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2fb00096c0 tx=0x7f2fb0005a20 comp rx=0 tx=0).stop 2024-09-10T03:36:26.666 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.664+0000 7f2fc1989640 1 -- 172.21.15.112:0/1099974635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2fbc100860 msgr2=0x7f2fbc104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:26.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.664+0000 7f2fc1989640 1 --2- 172.21.15.112:0/1099974635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2fbc100860 0x7f2fbc104780 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2fac00c8c0 tx=0x7f2fac00cd80 comp rx=0 tx=0).stop 2024-09-10T03:36:26.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.664+0000 7f2fc1989640 1 -- 172.21.15.112:0/1099974635 shutdown_connections 2024-09-10T03:36:26.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.664+0000 7f2fc1989640 1 --2- 172.21.15.112:0/1099974635 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2fa406ddb0 0x7f2fa4070270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:26.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.664+0000 7f2fc1989640 1 --2- 172.21.15.112:0/1099974635 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2fbc101a60 0x7f2fbc102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:26.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.664+0000 7f2fc1989640 1 --2- 172.21.15.112:0/1099974635 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2fbc100860 0x7f2fbc104780 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:26.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.664+0000 7f2fc1989640 1 -- 172.21.15.112:0/1099974635 >> 172.21.15.112:0/1099974635 conn(0x7f2fbc0fbff0 msgr2=0x7f2fbc06a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:36:26.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.664+0000 7f2fc1989640 1 -- 172.21.15.112:0/1099974635 shutdown_connections 2024-09-10T03:36:26.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.664+0000 7f2fc1989640 1 -- 172.21.15.112:0/1099974635 wait complete. 2024-09-10T03:36:26.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.683+0000 7f5919f73640 1 -- 172.21.15.112:0/697069817 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5914100880 msgr2=0x7f5914100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:26.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.683+0000 7f5919f73640 1 --2- 172.21.15.112:0/697069817 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5914100880 0x7f5914100c80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5904009880 tx=0x7f590402a4a0 comp rx=0 tx=0).stop 2024-09-10T03:36:26.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.683+0000 7f5919f73640 1 -- 172.21.15.112:0/697069817 shutdown_connections 2024-09-10T03:36:26.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.683+0000 7f5919f73640 1 --2- 172.21.15.112:0/697069817 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5914101a80 0x7f5914101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:26.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.683+0000 7f5919f73640 1 --2- 172.21.15.112:0/697069817 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5914100880 0x7f5914100c80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:26.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.683+0000 7f5919f73640 1 -- 172.21.15.112:0/697069817 >> 172.21.15.112:0/697069817 conn(0x7f59140fc010 msgr2=0x7f59140fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:36:26.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.683+0000 7f5919f73640 1 -- 172.21.15.112:0/697069817 shutdown_connections 2024-09-10T03:36:26.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.683+0000 7f5919f73640 1 -- 172.21.15.112:0/697069817 wait complete. 2024-09-10T03:36:26.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.683+0000 7f5919f73640 1 Processor -- start 2024-09-10T03:36:26.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.683+0000 7f5919f73640 1 -- start start 2024-09-10T03:36:26.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.684+0000 7f5919f73640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5914100880 0x7f591406ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:36:26.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.684+0000 7f5919f73640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5914101a80 0x7f591419c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:36:26.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.684+0000 7f5919f73640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f59141096a0 con 0x7f5914100880 2024-09-10T03:36:26.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.684+0000 7f5919f73640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f5914109520 con 0x7f5914101a80 2024-09-10T03:36:26.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.684+0000 7f59137fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5914100880 0x7f591406ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:36:26.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.684+0000 7f59137fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5914100880 0x7f591406ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52608/0 (socket says 172.21.15.112:52608) 2024-09-10T03:36:26.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.684+0000 7f59137fe640 1 -- 172.21.15.112:0/237650492 learned_addr learned my addr 172.21.15.112:0/237650492 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:36:26.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.684+0000 7f5912ffd640 1 --2- 172.21.15.112:0/237650492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5914101a80 0x7f591419c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:36:26.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.684+0000 7f59137fe640 1 -- 172.21.15.112:0/237650492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5914101a80 msgr2=0x7f591419c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:26.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.684+0000 7f59137fe640 1 --2- 172.21.15.112:0/237650492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5914101a80 0x7f591419c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:26.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.684+0000 7f59137fe640 1 -- 172.21.15.112:0/237650492 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f591419c7b0 con 0x7f5914100880 2024-09-10T03:36:26.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.684+0000 7f59137fe640 1 --2- 172.21.15.112:0/237650492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5914100880 0x7f591406ae40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5904009230 tx=0x7f59040048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:36:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.684+0000 7f5910ff9640 1 -- 172.21.15.112:0/237650492 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f590403f070 con 0x7f5914100880 2024-09-10T03:36:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.684+0000 7f5919f73640 1 -- 172.21.15.112:0/237650492 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f591419ca40 con 0x7f5914100880 2024-09-10T03:36:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.685+0000 7f5910ff9640 1 -- 172.21.15.112:0/237650492 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f59040044d0 con 0x7f5914100880 2024-09-10T03:36:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.685+0000 7f5910ff9640 1 -- 172.21.15.112:0/237650492 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f5904035030 con 0x7f5914100880 2024-09-10T03:36:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.685+0000 7f5919f73640 1 -- 172.21.15.112:0/237650492 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f591419cf70 con 0x7f5914100880 2024-09-10T03:36:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.686+0000 7f5910ff9640 1 -- 172.21.15.112:0/237650492 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 18) ==== 92149+0+0 (secure 0 0 0) 0x7f5904042070 con 0x7f5914100880 2024-09-10T03:36:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.686+0000 7f5919f73640 1 -- 172.21.15.112:0/237650492 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5914100c80 con 0x7f5914100880 2024-09-10T03:36:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.686+0000 7f5910ff9640 1 --2- 172.21.15.112:0/237650492 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f58fc06ddc0 0x7f58fc070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:36:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.687+0000 7f5910ff9640 1 -- 172.21.15.112:0/237650492 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f59040ad720 con 0x7f5914100880 2024-09-10T03:36:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.687+0000 7f5912ffd640 1 --2- 172.21.15.112:0/237650492 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f58fc06ddc0 0x7f58fc070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:36:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.687+0000 7f5912ffd640 1 --2- 172.21.15.112:0/237650492 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f58fc06ddc0 0x7f58fc070280 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f591406abc0 tx=0x7f590800a2d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:36:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.692+0000 7f5910ff9640 1 -- 172.21.15.112:0/237650492 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f590407ffc0 con 0x7f5914100880 2024-09-10T03:36:26.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.860+0000 7f5919f73640 1 -- 172.21.15.112:0/237650492 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f59141904b0 con 0x7f5914100880 2024-09-10T03:36:26.866 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.866+0000 7f5910ff9640 1 -- 172.21.15.112:0/237650492 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v28)=0 v28) ==== 121+0+0 (secure 0 0 0) 0x7f59040847f0 con 0x7f5914100880 2024-09-10T03:36:26.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.868+0000 7f5919f73640 1 -- 172.21.15.112:0/237650492 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f58fc06ddc0 msgr2=0x7f58fc070280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:26.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.868+0000 7f5919f73640 1 --2- 172.21.15.112:0/237650492 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f58fc06ddc0 0x7f58fc070280 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f591406abc0 tx=0x7f590800a2d0 comp rx=0 tx=0).stop 2024-09-10T03:36:26.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.868+0000 7f5919f73640 1 -- 172.21.15.112:0/237650492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5914100880 msgr2=0x7f591406ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:26.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.868+0000 7f5919f73640 1 --2- 172.21.15.112:0/237650492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5914100880 0x7f591406ae40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5904009230 tx=0x7f59040048f0 comp rx=0 tx=0).stop 2024-09-10T03:36:26.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.868+0000 7f5919f73640 1 -- 172.21.15.112:0/237650492 shutdown_connections 2024-09-10T03:36:26.869 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.868+0000 7f5919f73640 1 --2- 172.21.15.112:0/237650492 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f58fc06ddc0 0x7f58fc070280 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:26.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.868+0000 7f5919f73640 1 --2- 172.21.15.112:0/237650492 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5914101a80 0x7f591419c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:26.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.868+0000 7f5919f73640 1 --2- 172.21.15.112:0/237650492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5914100880 0x7f591406ae40 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:26.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.868+0000 7f5919f73640 1 -- 172.21.15.112:0/237650492 >> 172.21.15.112:0/237650492 conn(0x7f59140fc010 msgr2=0x7f5914104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:36:26.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.868+0000 7f5919f73640 1 -- 172.21.15.112:0/237650492 shutdown_connections 2024-09-10T03:36:26.870 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:26.868+0000 7f5919f73640 1 -- 172.21.15.112:0/237650492 wait complete. 2024-09-10T03:36:27.471 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:27 smithi112 ceph-mon[27737]: Reconfiguring crash.smithi112 (monmap changed)... 2024-09-10T03:36:27.471 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:27 smithi112 ceph-mon[27737]: Reconfiguring daemon crash.smithi112 on smithi112 2024-09-10T03:36:27.471 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:27 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/1099974635' entity='client.admin' 2024-09-10T03:36:27.471 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:27 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/237650492' entity='client.admin' 2024-09-10T03:36:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:27 smithi169 ceph-mon[35668]: Reconfiguring crash.smithi112 (monmap changed)... 2024-09-10T03:36:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:27 smithi169 ceph-mon[35668]: Reconfiguring daemon crash.smithi112 on smithi112 2024-09-10T03:36:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:27 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/1099974635' entity='client.admin' 2024-09-10T03:36:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:27 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/237650492' entity='client.admin' 2024-09-10T03:36:27.971 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi112') 2024-09-10T03:36:27.972 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi112') 2024-09-10T03:36:27.972 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi112') 2024-09-10T03:36:27.972 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi112') 2024-09-10T03:36:27.972 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi112') 2024-09-10T03:36:27.972 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi112') 2024-09-10T03:36:27.972 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi112') 2024-09-10T03:36:27.972 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi112') 2024-09-10T03:36:28.327 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:28 smithi112 ceph-mon[27737]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:28.358 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi112') 2024-09-10T03:36:28.358 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi112') 2024-09-10T03:36:28.359 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi112') 2024-09-10T03:36:28.359 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '513b3bc6-6f25-11ef-bcea-c7b262605968', '--', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi112') 2024-09-10T03:36:28.359 INFO:tasks.cephadm:Deploying OSDs... 2024-09-10T03:36:28.360 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T03:36:28.360 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T03:36:28.379 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T03:36:28.380 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_1 2024-09-10T03:36:28.436 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T03:36:28.436 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T03:36:28.436 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-10T03:36:28.437 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T03:36:28.437 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T03:36:28.437 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-10 03:35:28.412654648 +0000 2024-09-10T03:36:28.437 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-10 03:29:42.957826162 +0000 2024-09-10T03:36:28.437 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-10 03:29:42.957826162 +0000 2024-09-10T03:36:28.437 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-09-10 03:29:42.957826162 +0000 2024-09-10T03:36:28.437 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T03:36:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-10T03:36:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-10T03:36:28.503 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000215294 s, 2.4 MB/s 2024-09-10T03:36:28.504 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T03:36:28.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:28 smithi169 ceph-mon[35668]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:28.562 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_2 2024-09-10T03:36:28.618 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T03:36:28.618 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T03:36:28.618 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-10T03:36:28.618 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T03:36:28.618 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T03:36:28.619 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-10 03:35:28.413654620 +0000 2024-09-10T03:36:28.619 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-10 03:29:43.290817070 +0000 2024-09-10T03:36:28.619 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-10 03:29:43.290817070 +0000 2024-09-10T03:36:28.619 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-09-10 03:29:43.290817070 +0000 2024-09-10T03:36:28.619 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T03:36:28.684 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-10T03:36:28.684 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-10T03:36:28.684 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000194208 s, 2.6 MB/s 2024-09-10T03:36:28.685 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T03:36:28.742 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_3 2024-09-10T03:36:28.798 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T03:36:28.798 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T03:36:28.798 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-10T03:36:28.799 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T03:36:28.799 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T03:36:28.799 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-10 03:35:28.413654620 +0000 2024-09-10T03:36:28.799 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-10 03:29:43.663806886 +0000 2024-09-10T03:36:28.799 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-10 03:29:43.663806886 +0000 2024-09-10T03:36:28.799 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-09-10 03:29:43.663806886 +0000 2024-09-10T03:36:28.799 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T03:36:28.862 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-10T03:36:28.862 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-10T03:36:28.862 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000198558 s, 2.6 MB/s 2024-09-10T03:36:28.863 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T03:36:28.921 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_4 2024-09-10T03:36:28.976 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T03:36:28.977 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T03:36:28.977 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-10T03:36:28.977 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T03:36:28.977 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T03:36:28.977 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-10 03:35:28.414654593 +0000 2024-09-10T03:36:28.977 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-10 03:29:43.987798041 +0000 2024-09-10T03:36:28.977 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-10 03:29:43.987798041 +0000 2024-09-10T03:36:28.977 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-09-10 03:29:43.987798041 +0000 2024-09-10T03:36:28.977 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T03:36:29.040 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-10T03:36:29.040 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-10T03:36:29.040 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000171774 s, 3.0 MB/s 2024-09-10T03:36:29.041 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T03:36:29.098 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-10T03:36:29.098 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T03:36:29.122 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T03:36:29.122 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_1 2024-09-10T03:36:29.181 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T03:36:29.181 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T03:36:29.181 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-10T03:36:29.181 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T03:36:29.181 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T03:36:29.181 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-09-10 03:35:56.972811607 +0000 2024-09-10T03:36:29.182 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-09-10 03:29:42.963207348 +0000 2024-09-10T03:36:29.182 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-09-10 03:29:42.963207348 +0000 2024-09-10T03:36:29.182 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-09-10 03:29:42.963207348 +0000 2024-09-10T03:36:29.182 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T03:36:29.252 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-09-10T03:36:29.253 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-09-10T03:36:29.253 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.0002372 s, 2.2 MB/s 2024-09-10T03:36:29.253 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T03:36:29.311 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_2 2024-09-10T03:36:29.369 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T03:36:29.369 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T03:36:29.369 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-10T03:36:29.369 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T03:36:29.369 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T03:36:29.369 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-09-10 03:35:56.972811607 +0000 2024-09-10T03:36:29.369 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-09-10 03:29:43.302199826 +0000 2024-09-10T03:36:29.369 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-09-10 03:29:43.302199826 +0000 2024-09-10T03:36:29.369 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-09-10 03:29:43.302199826 +0000 2024-09-10T03:36:29.370 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T03:36:29.435 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-09-10T03:36:29.435 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-09-10T03:36:29.435 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000239556 s, 2.1 MB/s 2024-09-10T03:36:29.437 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T03:36:29.495 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_3 2024-09-10T03:36:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T03:36:29.552 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T03:36:29.552 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-10T03:36:29.552 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T03:36:29.552 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T03:36:29.552 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-09-10 03:35:56.972811607 +0000 2024-09-10T03:36:29.553 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-09-10 03:29:43.623192703 +0000 2024-09-10T03:36:29.553 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-09-10 03:29:43.623192703 +0000 2024-09-10T03:36:29.553 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-09-10 03:29:43.623192703 +0000 2024-09-10T03:36:29.553 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T03:36:29.617 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-09-10T03:36:29.618 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-09-10T03:36:29.618 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000228076 s, 2.2 MB/s 2024-09-10T03:36:29.619 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T03:36:29.677 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_4 2024-09-10T03:36:29.733 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T03:36:29.733 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T03:36:29.734 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-10T03:36:29.734 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T03:36:29.734 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T03:36:29.734 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-09-10 03:35:56.973811585 +0000 2024-09-10T03:36:29.734 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-09-10 03:29:43.942185624 +0000 2024-09-10T03:36:29.734 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-09-10 03:29:43.942185624 +0000 2024-09-10T03:36:29.734 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-09-10 03:29:43.942185624 +0000 2024-09-10T03:36:29.734 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T03:36:29.797 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-09-10T03:36:29.798 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-09-10T03:36:29.798 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000250822 s, 2.0 MB/s 2024-09-10T03:36:29.799 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T03:36:29.856 INFO:tasks.cephadm:Deploying osd.0 on smithi112 with /dev/vg_nvme/lv_4... 2024-09-10T03:36:29.856 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-10T03:36:30.300 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:30 smithi112 ceph-mon[27737]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:30.300 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:30 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:30.300 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:30 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:30.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:30 smithi169 ceph-mon[35668]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:30.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:30 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:30.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:30 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:31.451 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:31 smithi112 ceph-mon[27737]: Reconfiguring alertmanager.smithi112 (dependencies changed)... 2024-09-10T03:36:31.452 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:31 smithi112 ceph-mon[27737]: Reconfiguring daemon alertmanager.smithi112 on smithi112 2024-09-10T03:36:31.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:31 smithi169 ceph-mon[35668]: Reconfiguring alertmanager.smithi112 (dependencies changed)... 2024-09-10T03:36:31.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:31 smithi169 ceph-mon[35668]: Reconfiguring daemon alertmanager.smithi112 on smithi112 2024-09-10T03:36:32.128 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:32.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:32 smithi169 ceph-mon[35668]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:32.570 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:32 smithi112 ceph-mon[27737]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:34.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:34 smithi169 ceph-mon[35668]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:34.408 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:34 smithi112 ceph-mon[27737]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:35.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:36:35.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:36:36.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:36 smithi112 ceph-mon[27737]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:36.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:36 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:36.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:36 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:36.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:36 smithi169 ceph-mon[35668]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:36.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:36 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:36.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:36 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:37.085 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:36:37.111 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_4 2024-09-10T03:36:37.416 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:37 smithi112 ceph-mon[27737]: Reconfiguring grafana.smithi112 (dependencies changed)... 2024-09-10T03:36:37.416 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:37 smithi112 ceph-mon[27737]: Reconfiguring daemon grafana.smithi112 on smithi112 2024-09-10T03:36:37.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:37 smithi169 ceph-mon[35668]: Reconfiguring grafana.smithi112 (dependencies changed)... 2024-09-10T03:36:37.590 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:37 smithi169 ceph-mon[35668]: Reconfiguring daemon grafana.smithi112 on smithi112 2024-09-10T03:36:38.490 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:38 smithi112 ceph-mon[27737]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:38.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:38 smithi169 ceph-mon[35668]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:39.346 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:36:39.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.779+0000 7fa121af7640 1 -- 172.21.15.112:0/4114190002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa11c100880 msgr2=0x7fa11c100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:39.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.779+0000 7fa121af7640 1 --2- 172.21.15.112:0/4114190002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa11c100880 0x7fa11c100c80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa10c009880 tx=0x7fa10c02a4a0 comp rx=0 tx=0).stop 2024-09-10T03:36:39.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.779+0000 7fa121af7640 1 -- 172.21.15.112:0/4114190002 shutdown_connections 2024-09-10T03:36:39.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.779+0000 7fa121af7640 1 --2- 172.21.15.112:0/4114190002 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa11c101a80 0x7fa11c101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:39.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.779+0000 7fa121af7640 1 --2- 172.21.15.112:0/4114190002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa11c100880 0x7fa11c100c80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:39.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.779+0000 7fa121af7640 1 -- 172.21.15.112:0/4114190002 >> 172.21.15.112:0/4114190002 conn(0x7fa11c0fc010 msgr2=0x7fa11c0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:36:39.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.779+0000 7fa121af7640 1 -- 172.21.15.112:0/4114190002 shutdown_connections 2024-09-10T03:36:39.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.779+0000 7fa121af7640 1 -- 172.21.15.112:0/4114190002 wait complete. 2024-09-10T03:36:39.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.780+0000 7fa121af7640 1 Processor -- start 2024-09-10T03:36:39.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.780+0000 7fa121af7640 1 -- start start 2024-09-10T03:36:39.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.780+0000 7fa121af7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa11c100880 0x7fa11c06adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:36:39.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.780+0000 7fa121af7640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa11c101a80 0x7fa11c19c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:36:39.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.780+0000 7fa121af7640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c1096c0 con 0x7fa11c101a80 2024-09-10T03:36:39.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.780+0000 7fa121af7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c109540 con 0x7fa11c100880 2024-09-10T03:36:39.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.780+0000 7fa11affd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa11c101a80 0x7fa11c19c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:36:39.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.780+0000 7fa11affd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa11c101a80 0x7fa11c19c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:41844/0 (socket says 172.21.15.112:41844) 2024-09-10T03:36:39.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.780+0000 7fa11affd640 1 -- 172.21.15.112:0/4220944689 learned_addr learned my addr 172.21.15.112:0/4220944689 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:36:39.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.780+0000 7fa11b7fe640 1 --2- 172.21.15.112:0/4220944689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa11c100880 0x7fa11c06adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:36:39.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.781+0000 7fa11affd640 1 -- 172.21.15.112:0/4220944689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa11c100880 msgr2=0x7fa11c06adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:36:39.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.781+0000 7fa11affd640 1 --2- 172.21.15.112:0/4220944689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa11c100880 0x7fa11c06adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:36:39.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.781+0000 7fa11affd640 1 -- 172.21.15.112:0/4220944689 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa11c19c7d0 con 0x7fa11c101a80 2024-09-10T03:36:39.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.781+0000 7fa11affd640 1 --2- 172.21.15.112:0/4220944689 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa11c101a80 0x7fa11c19c210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa10400d6f0 tx=0x7fa10400dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:36:39.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.781+0000 7fa118ff9640 1 -- 172.21.15.112:0/4220944689 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fa104017070 con 0x7fa11c101a80 2024-09-10T03:36:39.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.781+0000 7fa118ff9640 1 -- 172.21.15.112:0/4220944689 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa10400de00 con 0x7fa11c101a80 2024-09-10T03:36:39.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.781+0000 7fa121af7640 1 -- 172.21.15.112:0/4220944689 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa11c19cac0 con 0x7fa11c101a80 2024-09-10T03:36:39.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.781+0000 7fa121af7640 1 -- 172.21.15.112:0/4220944689 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa11c19d050 con 0x7fa11c101a80 2024-09-10T03:36:39.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.781+0000 7fa118ff9640 1 -- 172.21.15.112:0/4220944689 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fa104004bb0 con 0x7fa11c101a80 2024-09-10T03:36:39.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.782+0000 7fa121af7640 1 -- 172.21.15.112:0/4220944689 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa11c100c80 con 0x7fa11c101a80 2024-09-10T03:36:39.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.783+0000 7fa118ff9640 1 -- 172.21.15.112:0/4220944689 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 18) ==== 92149+0+0 (secure 0 0 0) 0x7fa104009a30 con 0x7fa11c101a80 2024-09-10T03:36:39.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.784+0000 7fa118ff9640 1 --2- 172.21.15.112:0/4220944689 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa10806de00 0x7fa1080702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:36:39.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.784+0000 7fa118ff9640 1 -- 172.21.15.112:0/4220944689 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fa10408c650 con 0x7fa11c101a80 2024-09-10T03:36:39.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.784+0000 7fa11b7fe640 1 --2- 172.21.15.112:0/4220944689 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa10806de00 0x7fa1080702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:36:39.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.784+0000 7fa11b7fe640 1 --2- 172.21.15.112:0/4220944689 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa10806de00 0x7fa1080702c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa10c009230 tx=0x7fa10c0058d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:36:39.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.788+0000 7fa118ff9640 1 -- 172.21.15.112:0/4220944689 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa10405ef10 con 0x7fa11c101a80 2024-09-10T03:36:39.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:36:39.971+0000 7fa121af7640 1 -- 172.21.15.112:0/4220944689 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fa11c106000 con 0x7fa10806de00 2024-09-10T03:36:40.277 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:40 smithi112 ceph-mon[27737]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:40.277 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T03:36:40.277 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T03:36:40.277 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:40.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:40 smithi169 ceph-mon[35668]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:40.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:40 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T03:36:40.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:40 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T03:36:40.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:40 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:41.292 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:41 smithi112 ceph-mon[27737]: from='client.14332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:36:41.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:41 smithi169 ceph-mon[35668]: from='client.14332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:36:42.248 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:42 smithi112 ceph-mon[27737]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:42.427 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:42 smithi169 ceph-mon[35668]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:43.695 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:43 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:43.695 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:43 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:43.695 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:43 smithi169 ceph-mon[35668]: Reconfiguring prometheus.smithi112 (dependencies changed)... 2024-09-10T03:36:43.695 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:43 smithi169 ceph-mon[35668]: Reconfiguring daemon prometheus.smithi112 on smithi112 2024-09-10T03:36:43.695 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:43 smithi169 ceph-mon[35668]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:43.741 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:43 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:43.741 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:43 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:43.741 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:43 smithi112 ceph-mon[27737]: Reconfiguring prometheus.smithi112 (dependencies changed)... 2024-09-10T03:36:43.741 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:43 smithi112 ceph-mon[27737]: Reconfiguring daemon prometheus.smithi112 on smithi112 2024-09-10T03:36:43.741 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:43 smithi112 ceph-mon[27737]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:44.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:44 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3386202899' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae9a7fcc-9d61-4914-b88f-da4fc76d5b88"}]: dispatch 2024-09-10T03:36:44.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:44 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3386202899' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae9a7fcc-9d61-4914-b88f-da4fc76d5b88"}]': finished 2024-09-10T03:36:44.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:44 smithi169 ceph-mon[35668]: osdmap e6: 1 total, 0 up, 1 in 2024-09-10T03:36:44.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:44 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T03:36:44.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:44 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/1593890553' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T03:36:44.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:44 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3386202899' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae9a7fcc-9d61-4914-b88f-da4fc76d5b88"}]: dispatch 2024-09-10T03:36:44.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:44 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3386202899' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae9a7fcc-9d61-4914-b88f-da4fc76d5b88"}]': finished 2024-09-10T03:36:44.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:44 smithi112 ceph-mon[27737]: osdmap e6: 1 total, 0 up, 1 in 2024-09-10T03:36:44.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:44 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T03:36:44.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:44 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/1593890553' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T03:36:45.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:45 smithi169 ceph-mon[35668]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:45.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:45 smithi112 ceph-mon[27737]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:48.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:47 smithi169 ceph-mon[35668]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:48.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:47 smithi112 ceph-mon[27737]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:50.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:49 smithi112 ceph-mon[27737]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:50.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:49 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:36:50.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:49 smithi169 ceph-mon[35668]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:50.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:49 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:36:52.004 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:51 smithi112 ceph-mon[27737]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:52.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:51 smithi169 ceph-mon[35668]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:54.160 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:53 smithi169 ceph-mon[35668]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:54.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:53 smithi112 ceph-mon[27737]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:55.984 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:55 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:55.985 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:55 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:55.985 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:55 smithi112 ceph-mon[27737]: Reconfiguring ceph-exporter.smithi169 (monmap changed)... 2024-09-10T03:36:55.985 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:55 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi169", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-10T03:36:55.985 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:55 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:55.985 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:55 smithi112 ceph-mon[27737]: Reconfiguring daemon ceph-exporter.smithi169 on smithi169 2024-09-10T03:36:55.985 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:55 smithi112 ceph-mon[27737]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:56.003 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:55 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:56.004 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:55 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:56.004 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:55 smithi169 ceph-mon[35668]: Reconfiguring ceph-exporter.smithi169 (monmap changed)... 2024-09-10T03:36:56.004 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:55 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi169", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-10T03:36:56.004 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:55 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:56.004 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:55 smithi169 ceph-mon[35668]: Reconfiguring daemon ceph-exporter.smithi169 on smithi169 2024-09-10T03:36:56.004 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:55 smithi169 ceph-mon[35668]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:57.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:56 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-10T03:36:57.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:56 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:57.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:56 smithi169 ceph-mon[35668]: Deploying daemon osd.0 on smithi112 2024-09-10T03:36:57.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:56 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:57.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:56 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:57.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:56 smithi169 ceph-mon[35668]: Reconfiguring crash.smithi169 (monmap changed)... 2024-09-10T03:36:57.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:56 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi169", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-10T03:36:57.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:56 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:57.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:56 smithi169 ceph-mon[35668]: Reconfiguring daemon crash.smithi169 on smithi169 2024-09-10T03:36:57.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:56 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-10T03:36:57.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:56 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:57.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:56 smithi112 ceph-mon[27737]: Deploying daemon osd.0 on smithi112 2024-09-10T03:36:57.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:56 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:57.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:56 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:57.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:56 smithi112 ceph-mon[27737]: Reconfiguring crash.smithi169 (monmap changed)... 2024-09-10T03:36:57.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:56 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi169", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-10T03:36:57.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:56 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:57.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:56 smithi112 ceph-mon[27737]: Reconfiguring daemon crash.smithi169 on smithi169 2024-09-10T03:36:57.810 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:57 smithi112 ceph-mon[27737]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:57.810 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:57 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:57.810 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:57 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:57.811 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:57 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi169.bkwgjl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T03:36:57.811 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:57 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T03:36:57.811 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:57 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:58.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:57 smithi169 ceph-mon[35668]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:36:58.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:58.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:36:58.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi169.bkwgjl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T03:36:58.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T03:36:58.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:57 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:36:58.973 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:58 smithi169 ceph-mon[35668]: Reconfiguring mgr.smithi169.bkwgjl (monmap changed)... 2024-09-10T03:36:58.973 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:58 smithi169 ceph-mon[35668]: Reconfiguring daemon mgr.smithi169.bkwgjl on smithi169 2024-09-10T03:36:58.991 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:58 smithi112 ceph-mon[27737]: Reconfiguring mgr.smithi169.bkwgjl (monmap changed)... 2024-09-10T03:36:58.991 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:58 smithi112 ceph-mon[27737]: Reconfiguring daemon mgr.smithi169.bkwgjl on smithi169 2024-09-10T03:37:00.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:59 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:00.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:59 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:00.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:59 smithi169 ceph-mon[35668]: Reconfiguring mon.smithi169 (monmap changed)... 2024-09-10T03:37:00.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:59 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T03:37:00.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:59 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T03:37:00.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:59 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:00.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:59 smithi169 ceph-mon[35668]: Reconfiguring daemon mon.smithi169 on smithi169 2024-09-10T03:37:00.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:36:59 smithi169 ceph-mon[35668]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:37:00.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:59 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:00.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:59 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:00.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:59 smithi112 ceph-mon[27737]: Reconfiguring mon.smithi169 (monmap changed)... 2024-09-10T03:37:00.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:59 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T03:37:00.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:59 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T03:37:00.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:59 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:00.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:59 smithi112 ceph-mon[27737]: Reconfiguring daemon mon.smithi169 on smithi169 2024-09-10T03:37:00.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:36:59 smithi112 ceph-mon[27737]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:37:01.369 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:01.369 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:01.370 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T03:37:01.370 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T03:37:01.370 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi112.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T03:37:01.370 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi112.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T03:37:01.370 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:01.370 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T03:37:01.370 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T03:37:01.370 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi112.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T03:37:01.370 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi112.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T03:37:01.370 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:01.370 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T03:37:01.370 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T03:37:01.371 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi112.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T03:37:01.371 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi112.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T03:37:01.371 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:01.371 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:01 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:37:01.459 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:01.459 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:01.459 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T03:37:01.459 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T03:37:01.460 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi112.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T03:37:01.460 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi112.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T03:37:01.460 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:01.460 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T03:37:01.460 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T03:37:01.460 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi112.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T03:37:01.460 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi112.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T03:37:01.460 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:01.460 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T03:37:01.460 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T03:37:01.460 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi112.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T03:37:01.460 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi112.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T03:37:01.461 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:01.461 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:01 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:37:02.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:02 smithi112 ceph-mon[27737]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:37:02.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:02 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:02.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:02 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:02.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:02 smithi169 ceph-mon[35668]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:37:02.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:02 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:02.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:02 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:03.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:03 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:03.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:03 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:03.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:03 smithi112 ceph-mon[27737]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:37:03.974 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:03 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:03.974 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:03 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:03.975 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:03 smithi169 ceph-mon[35668]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:37:05.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:04 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:37:05.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:04 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:37:06.181 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:05 smithi112 ceph-mon[27737]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:37:06.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:05 smithi169 ceph-mon[35668]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:37:08.019 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 0 on host 'smithi112' 2024-09-10T03:37:08.019 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:08.016+0000 7fa118ff9640 1 -- 172.21.15.112:0/4220944689 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa11c106000 con 0x7fa10806de00 2024-09-10T03:37:08.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:08.019+0000 7fa121af7640 1 -- 172.21.15.112:0/4220944689 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa10806de00 msgr2=0x7fa1080702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:37:08.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:08.019+0000 7fa121af7640 1 --2- 172.21.15.112:0/4220944689 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa10806de00 0x7fa1080702c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa10c009230 tx=0x7fa10c0058d0 comp rx=0 tx=0).stop 2024-09-10T03:37:08.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:08.019+0000 7fa121af7640 1 -- 172.21.15.112:0/4220944689 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa11c101a80 msgr2=0x7fa11c19c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:37:08.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:08.019+0000 7fa121af7640 1 --2- 172.21.15.112:0/4220944689 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa11c101a80 0x7fa11c19c210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa10400d6f0 tx=0x7fa10400dbb0 comp rx=0 tx=0).stop 2024-09-10T03:37:08.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:08.019+0000 7fa121af7640 1 -- 172.21.15.112:0/4220944689 shutdown_connections 2024-09-10T03:37:08.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:08.019+0000 7fa121af7640 1 --2- 172.21.15.112:0/4220944689 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa10806de00 0x7fa1080702c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:37:08.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:08.019+0000 7fa121af7640 1 --2- 172.21.15.112:0/4220944689 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa11c101a80 0x7fa11c19c210 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:37:08.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:08.019+0000 7fa121af7640 1 --2- 172.21.15.112:0/4220944689 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa11c100880 0x7fa11c06adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:37:08.021 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:08.019+0000 7fa121af7640 1 -- 172.21.15.112:0/4220944689 >> 172.21.15.112:0/4220944689 conn(0x7fa11c0fc010 msgr2=0x7fa11c0fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:37:08.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:08.019+0000 7fa121af7640 1 -- 172.21.15.112:0/4220944689 shutdown_connections 2024-09-10T03:37:08.022 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:08.019+0000 7fa121af7640 1 -- 172.21.15.112:0/4220944689 wait complete. 2024-09-10T03:37:08.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:08 smithi112 ceph-mon[27737]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:37:08.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:08 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:08.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:08 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:08.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:08 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:08.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:08 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:37:08.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:08 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:08.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:08 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:37:08.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:08 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:08.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:08 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:08.372 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:08 smithi169 ceph-mon[35668]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:37:08.372 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:08 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:08.372 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:08 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:08.372 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:08 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:08.372 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:08 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:37:08.372 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:08 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:08.372 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:08 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:37:08.372 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:08 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:08.372 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:08 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:08.643 DEBUG:teuthology.orchestra.run.smithi112:osd.0> sudo journalctl -f -n 0 -u ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@osd.0.service 2024-09-10T03:37:08.645 INFO:tasks.cephadm:Deploying osd.1 on smithi112 with /dev/vg_nvme/lv_3... 2024-09-10T03:37:08.645 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-10T03:37:09.021 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:37:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:37:08.743+0000 7f9b67ae8640 -1 osd.0 0 log_to_monitors true 2024-09-10T03:37:09.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:09 smithi169 ceph-mon[35668]: from='osd.0 [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T03:37:09.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:09 smithi112 ceph-mon[27737]: from='osd.0 [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T03:37:09.517 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:37:10.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:10 smithi112 ceph-mon[27737]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:37:10.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:10 smithi112 ceph-mon[27737]: from='osd.0 [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-10T03:37:10.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:10 smithi112 ceph-mon[27737]: osdmap e7: 1 total, 0 up, 1 in 2024-09-10T03:37:10.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:10 smithi112 ceph-mon[27737]: from='osd.0 [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-09-10T03:37:10.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:10 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T03:37:10.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:10 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:10.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:10 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:10.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:10 smithi169 ceph-mon[35668]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:37:10.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:10 smithi169 ceph-mon[35668]: from='osd.0 [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-10T03:37:10.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:10 smithi169 ceph-mon[35668]: osdmap e7: 1 total, 0 up, 1 in 2024-09-10T03:37:10.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:10 smithi169 ceph-mon[35668]: from='osd.0 [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-09-10T03:37:10.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:10 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T03:37:10.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:10 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:10.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:10 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:11.034 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:37:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:37:10.888+0000 7f9b63248640 -1 osd.0 0 waiting for initial osdmap 2024-09-10T03:37:11.034 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:37:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:37:10.892+0000 7f9b5b87a640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T03:37:11.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:11 smithi112 ceph-mon[27737]: from='osd.0 [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-09-10T03:37:11.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:11 smithi112 ceph-mon[27737]: osdmap e8: 1 total, 0 up, 1 in 2024-09-10T03:37:11.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:11 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T03:37:11.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:11 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T03:37:11.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:11 smithi112 ceph-mon[27737]: from='osd.0 [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782]' entity='osd.0' 2024-09-10T03:37:11.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:11 smithi169 ceph-mon[35668]: from='osd.0 [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-09-10T03:37:11.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:11 smithi169 ceph-mon[35668]: osdmap e8: 1 total, 0 up, 1 in 2024-09-10T03:37:11.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:11 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T03:37:11.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:11 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T03:37:11.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:11 smithi169 ceph-mon[35668]: from='osd.0 [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782]' entity='osd.0' 2024-09-10T03:37:12.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:12 smithi112 ceph-mon[27737]: purged_snaps scrub starts 2024-09-10T03:37:12.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:12 smithi112 ceph-mon[27737]: purged_snaps scrub ok 2024-09-10T03:37:12.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:12 smithi112 ceph-mon[27737]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:37:12.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:12 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T03:37:12.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:12 smithi112 ceph-mon[27737]: osd.0 [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782] boot 2024-09-10T03:37:12.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:12 smithi112 ceph-mon[27737]: osdmap e9: 1 total, 1 up, 1 in 2024-09-10T03:37:12.285 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:12 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T03:37:12.427 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:12 smithi169 ceph-mon[35668]: purged_snaps scrub starts 2024-09-10T03:37:12.427 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:12 smithi169 ceph-mon[35668]: purged_snaps scrub ok 2024-09-10T03:37:12.427 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:12 smithi169 ceph-mon[35668]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T03:37:12.428 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:12 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T03:37:12.428 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:12 smithi169 ceph-mon[35668]: osd.0 [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782] boot 2024-09-10T03:37:12.428 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:12 smithi169 ceph-mon[35668]: osdmap e9: 1 total, 1 up, 1 in 2024-09-10T03:37:12.428 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:12 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T03:37:12.909 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:37:12.934 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_3 2024-09-10T03:37:13.681 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:37:14.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:14 smithi169 ceph-mon[35668]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:14.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:14 smithi169 ceph-mon[35668]: osdmap e10: 1 total, 1 up, 1 in 2024-09-10T03:37:14.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.337+0000 7fa4ec5b5640 1 -- 172.21.15.112:0/2066071377 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e4102b90 msgr2=0x7fa4e4103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:37:14.338 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.337+0000 7fa4ec5b5640 1 --2- 172.21.15.112:0/2066071377 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e4102b90 0x7fa4e4103010 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8009880 tx=0x7fa4d802a4a0 comp rx=0 tx=0).stop 2024-09-10T03:37:14.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.337+0000 7fa4ec5b5640 1 -- 172.21.15.112:0/2066071377 shutdown_connections 2024-09-10T03:37:14.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.337+0000 7fa4ec5b5640 1 --2- 172.21.15.112:0/2066071377 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e4102b90 0x7fa4e4103010 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:37:14.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.337+0000 7fa4ec5b5640 1 --2- 172.21.15.112:0/2066071377 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4e4101990 0x7fa4e4101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:37:14.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.337+0000 7fa4ec5b5640 1 -- 172.21.15.112:0/2066071377 >> 172.21.15.112:0/2066071377 conn(0x7fa4e40fd140 msgr2=0x7fa4e40ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:37:14.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.337+0000 7fa4ec5b5640 1 -- 172.21.15.112:0/2066071377 shutdown_connections 2024-09-10T03:37:14.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.337+0000 7fa4ec5b5640 1 -- 172.21.15.112:0/2066071377 wait complete. 2024-09-10T03:37:14.340 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.338+0000 7fa4ec5b5640 1 Processor -- start 2024-09-10T03:37:14.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.338+0000 7fa4ec5b5640 1 -- start start 2024-09-10T03:37:14.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.338+0000 7fa4ec5b5640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e4101990 0x7fa4e41937d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:37:14.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.338+0000 7fa4ec5b5640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4e4102b90 0x7fa4e4193d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:37:14.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.338+0000 7fa4ec5b5640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e410a7d0 con 0x7fa4e4101990 2024-09-10T03:37:14.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.338+0000 7fa4ec5b5640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fa4e410a650 con 0x7fa4e4102b90 2024-09-10T03:37:14.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.338+0000 7fa4ea351640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e4101990 0x7fa4e41937d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:37:14.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.338+0000 7fa4ea351640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e4101990 0x7fa4e41937d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:43708/0 (socket says 172.21.15.112:43708) 2024-09-10T03:37:14.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.338+0000 7fa4ea351640 1 -- 172.21.15.112:0/2066714705 learned_addr learned my addr 172.21.15.112:0/2066714705 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:37:14.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.338+0000 7fa4e9b50640 1 --2- 172.21.15.112:0/2066714705 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4e4102b90 0x7fa4e4193d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:37:14.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.338+0000 7fa4ea351640 1 -- 172.21.15.112:0/2066714705 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4e4102b90 msgr2=0x7fa4e4193d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:37:14.341 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.338+0000 7fa4ea351640 1 --2- 172.21.15.112:0/2066714705 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4e4102b90 0x7fa4e4193d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:37:14.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.338+0000 7fa4ea351640 1 -- 172.21.15.112:0/2066714705 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e4190350 con 0x7fa4e4101990 2024-09-10T03:37:14.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.339+0000 7fa4ea351640 1 --2- 172.21.15.112:0/2066714705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e4101990 0x7fa4e41937d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc00ea20 tx=0x7fa4dc00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:37:14.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.339+0000 7fa4e37fe640 1 -- 172.21.15.112:0/2066714705 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fa4dc016070 con 0x7fa4e4101990 2024-09-10T03:37:14.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.339+0000 7fa4e37fe640 1 -- 172.21.15.112:0/2066714705 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa4dc003e30 con 0x7fa4e4101990 2024-09-10T03:37:14.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.339+0000 7fa4ec5b5640 1 -- 172.21.15.112:0/2066714705 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4e4190640 con 0x7fa4e4101990 2024-09-10T03:37:14.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.339+0000 7fa4e37fe640 1 -- 172.21.15.112:0/2066714705 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fa4dc011470 con 0x7fa4e4101990 2024-09-10T03:37:14.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.339+0000 7fa4ec5b5640 1 -- 172.21.15.112:0/2066714705 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4e4190bd0 con 0x7fa4e4101990 2024-09-10T03:37:14.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.340+0000 7fa4ec5b5640 1 -- 172.21.15.112:0/2066714705 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4e4101d90 con 0x7fa4e4101990 2024-09-10T03:37:14.342 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.341+0000 7fa4e37fe640 1 -- 172.21.15.112:0/2066714705 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 18) ==== 92149+0+0 (secure 0 0 0) 0x7fa4dc019070 con 0x7fa4e4101990 2024-09-10T03:37:14.343 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:14 smithi112 ceph-mon[27737]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:14.343 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:14 smithi112 ceph-mon[27737]: osdmap e10: 1 total, 1 up, 1 in 2024-09-10T03:37:14.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.341+0000 7fa4e37fe640 1 --2- 172.21.15.112:0/2066714705 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa4cc06ddb0 0x7fa4cc070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:37:14.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.341+0000 7fa4e37fe640 1 -- 172.21.15.112:0/2066714705 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1982+0+0 (secure 0 0 0) 0x7fa4dc08cf80 con 0x7fa4e4101990 2024-09-10T03:37:14.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.342+0000 7fa4e9b50640 1 --2- 172.21.15.112:0/2066714705 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa4cc06ddb0 0x7fa4cc070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:37:14.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.342+0000 7fa4e9b50640 1 --2- 172.21.15.112:0/2066714705 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa4cc06ddb0 0x7fa4cc070270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8006010 tx=0x7fa4d8000bc0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:37:14.347 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.347+0000 7fa4e37fe640 1 -- 172.21.15.112:0/2066714705 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa4dc05b9a0 con 0x7fa4e4101990 2024-09-10T03:37:14.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:14.523+0000 7fa4ec5b5640 1 -- 172.21.15.112:0/2066714705 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fa4e402db20 con 0x7fa4cc06ddb0 2024-09-10T03:37:15.252 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:15 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T03:37:15.253 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:15 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T03:37:15.253 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:15 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:15.253 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:15 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:15.253 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:15 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:15.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:15 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T03:37:15.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:15 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T03:37:15.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:15 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:15.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:15 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:15.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:15 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:16.293 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:16 smithi112 ceph-mon[27737]: from='client.14350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:37:16.293 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:16 smithi112 ceph-mon[27737]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:16.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:16 smithi169 ceph-mon[35668]: from='client.14350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:37:16.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:16 smithi169 ceph-mon[35668]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:18.402 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:18 smithi112 ceph-mon[27737]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:18.402 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:18 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3401695402' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41b3eaeb-7456-42dd-9405-0770a7129bc8"}]: dispatch 2024-09-10T03:37:18.402 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:18 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3401695402' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41b3eaeb-7456-42dd-9405-0770a7129bc8"}]': finished 2024-09-10T03:37:18.402 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:18 smithi112 ceph-mon[27737]: osdmap e11: 2 total, 1 up, 2 in 2024-09-10T03:37:18.402 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:18 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T03:37:18.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:18 smithi169 ceph-mon[35668]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:18.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:18 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3401695402' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41b3eaeb-7456-42dd-9405-0770a7129bc8"}]: dispatch 2024-09-10T03:37:18.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:18 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3401695402' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41b3eaeb-7456-42dd-9405-0770a7129bc8"}]': finished 2024-09-10T03:37:18.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:18 smithi169 ceph-mon[35668]: osdmap e11: 2 total, 1 up, 2 in 2024-09-10T03:37:18.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:18 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T03:37:19.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:19 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/2498996857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T03:37:19.568 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:19 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2498996857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T03:37:20.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:20 smithi169 ceph-mon[35668]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:20.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:20 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:37:20.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:20 smithi112 ceph-mon[27737]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:20.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:20 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:37:21.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:21 smithi169 ceph-mon[35668]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:21.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:21 smithi112 ceph-mon[27737]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:24.029 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:23 smithi169 ceph-mon[35668]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:24.174 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:23 smithi112 ceph-mon[27737]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:26.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:25 smithi112 ceph-mon[27737]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:26.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:25 smithi169 ceph-mon[35668]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:27.465 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:27 smithi112 ceph-mon[27737]: Detected new or changed devices on smithi112 2024-09-10T03:37:27.466 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:27 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:27.466 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:27 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:27.466 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:27 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:37:27.466 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:27 smithi112 ceph-mon[27737]: Adjusting osd_memory_target on smithi112 to 11840M 2024-09-10T03:37:27.466 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:27 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:27.466 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:27 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:27.466 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:27 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:37:27.466 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:27 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:27.466 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:27 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:37:27.466 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:27 smithi112 ceph-mon[27737]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:27 smithi169 ceph-mon[35668]: Detected new or changed devices on smithi112 2024-09-10T03:37:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:27 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:27 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:27 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:37:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:27 smithi169 ceph-mon[35668]: Adjusting osd_memory_target on smithi112 to 11840M 2024-09-10T03:37:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:27 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:27 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:27 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:37:27.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:27 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:27.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:27 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:37:27.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:27 smithi169 ceph-mon[35668]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:28.529 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:28 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-10T03:37:28.529 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:28 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:28.529 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:28 smithi169 ceph-mon[35668]: Deploying daemon osd.1 on smithi112 2024-09-10T03:37:28.529 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:28 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:28.529 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:28 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:28.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:28 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-10T03:37:28.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:28 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:28.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:28 smithi112 ceph-mon[27737]: Deploying daemon osd.1 on smithi112 2024-09-10T03:37:28.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:28 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:28.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:28 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:29.416 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:29 smithi112 ceph-mon[27737]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:29.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:29 smithi169 ceph-mon[35668]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:32.251 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:31 smithi112 ceph-mon[27737]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:32.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:31 smithi169 ceph-mon[35668]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:34.212 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:33 smithi112 ceph-mon[27737]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:34.212 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:33 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:34.212 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:33 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:34.212 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:33 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:34.213 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:33 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:37:34.213 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:33 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:34.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:33 smithi169 ceph-mon[35668]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:34.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:33 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:34.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:33 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:34.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:33 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:34.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:33 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:37:34.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:33 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:35.168 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:34 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:37:35.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:34 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:37:36.175 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:35 smithi112 ceph-mon[27737]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:36.175 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:37:36.175 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:36.175 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:36.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:35 smithi169 ceph-mon[35668]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:36.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:37:36.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:36.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:38.175 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:37 smithi112 ceph-mon[27737]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:38.175 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:37 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:38.175 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:37 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:38.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:37 smithi169 ceph-mon[35668]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:38.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:37 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:38.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:37 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:40.004 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:39 smithi112 ceph-mon[27737]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:40.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:39 smithi169 ceph-mon[35668]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:40.327 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:40.326+0000 7fa4e37fe640 1 -- 172.21.15.112:0/2066714705 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa4e402db20 con 0x7fa4cc06ddb0 2024-09-10T03:37:40.328 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 1 on host 'smithi112' 2024-09-10T03:37:40.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:40.328+0000 7fa4ec5b5640 1 -- 172.21.15.112:0/2066714705 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa4cc06ddb0 msgr2=0x7fa4cc070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:37:40.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:40.328+0000 7fa4ec5b5640 1 --2- 172.21.15.112:0/2066714705 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa4cc06ddb0 0x7fa4cc070270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8006010 tx=0x7fa4d8000bc0 comp rx=0 tx=0).stop 2024-09-10T03:37:40.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:40.328+0000 7fa4ec5b5640 1 -- 172.21.15.112:0/2066714705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e4101990 msgr2=0x7fa4e41937d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:37:40.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:40.328+0000 7fa4ec5b5640 1 --2- 172.21.15.112:0/2066714705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e4101990 0x7fa4e41937d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc00ea20 tx=0x7fa4dc00eee0 comp rx=0 tx=0).stop 2024-09-10T03:37:40.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:40.328+0000 7fa4ec5b5640 1 -- 172.21.15.112:0/2066714705 shutdown_connections 2024-09-10T03:37:40.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:40.328+0000 7fa4ec5b5640 1 --2- 172.21.15.112:0/2066714705 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa4cc06ddb0 0x7fa4cc070270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:37:40.329 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:40.328+0000 7fa4ec5b5640 1 --2- 172.21.15.112:0/2066714705 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4e4102b90 0x7fa4e4193d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:37:40.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:40.328+0000 7fa4ec5b5640 1 --2- 172.21.15.112:0/2066714705 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4e4101990 0x7fa4e41937d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:37:40.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:40.328+0000 7fa4ec5b5640 1 -- 172.21.15.112:0/2066714705 >> 172.21.15.112:0/2066714705 conn(0x7fa4e40fd140 msgr2=0x7fa4e4104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:37:40.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:40.328+0000 7fa4ec5b5640 1 -- 172.21.15.112:0/2066714705 shutdown_connections 2024-09-10T03:37:40.330 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:40.328+0000 7fa4ec5b5640 1 -- 172.21.15.112:0/2066714705 wait complete. 2024-09-10T03:37:41.026 DEBUG:teuthology.orchestra.run.smithi112:osd.1> sudo journalctl -f -n 0 -u ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@osd.1.service 2024-09-10T03:37:41.029 INFO:tasks.cephadm:Deploying osd.2 on smithi112 with /dev/vg_nvme/lv_2... 2024-09-10T03:37:41.030 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-10T03:37:41.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:41 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:41.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:41 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:41.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:41 smithi112 ceph-mon[27737]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:41.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:41 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:41.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:41 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:41.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:41 smithi169 ceph-mon[35668]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:42.328 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:37:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:37:42.034+0000 7f1db21d1640 -1 osd.1 0 log_to_monitors true 2024-09-10T03:37:42.510 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:37:42.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:42 smithi112 ceph-mon[27737]: from='osd.1 [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T03:37:42.677 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:42 smithi169 ceph-mon[35668]: from='osd.1 [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T03:37:43.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:43 smithi112 ceph-mon[27737]: from='osd.1 [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T03:37:43.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:43 smithi112 ceph-mon[27737]: osdmap e12: 2 total, 1 up, 2 in 2024-09-10T03:37:43.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:43 smithi112 ceph-mon[27737]: from='osd.1 [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-09-10T03:37:43.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:43 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T03:37:43.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:43 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:43.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:43 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:43.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:43 smithi112 ceph-mon[27737]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:43.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:43 smithi169 ceph-mon[35668]: from='osd.1 [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T03:37:43.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:43 smithi169 ceph-mon[35668]: osdmap e12: 2 total, 1 up, 2 in 2024-09-10T03:37:43.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:43 smithi169 ceph-mon[35668]: from='osd.1 [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-09-10T03:37:43.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:43 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T03:37:43.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:43 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:43.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:43 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:43.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:43 smithi169 ceph-mon[35668]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:37:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:37:44.115+0000 7f1dad931640 -1 osd.1 0 waiting for initial osdmap 2024-09-10T03:37:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:37:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:37:44.121+0000 7f1da5f63640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T03:37:44.635 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:44 smithi112 ceph-mon[27737]: from='osd.1 [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-09-10T03:37:44.635 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:44 smithi112 ceph-mon[27737]: osdmap e13: 2 total, 1 up, 2 in 2024-09-10T03:37:44.635 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:44 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T03:37:44.635 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:44 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T03:37:44.636 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:44 smithi112 ceph-mon[27737]: from='osd.1 [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264]' entity='osd.1' 2024-09-10T03:37:44.636 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:44 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T03:37:44.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:44 smithi169 ceph-mon[35668]: from='osd.1 [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-09-10T03:37:44.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:44 smithi169 ceph-mon[35668]: osdmap e13: 2 total, 1 up, 2 in 2024-09-10T03:37:44.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:44 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T03:37:44.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:44 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T03:37:44.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:44 smithi169 ceph-mon[35668]: from='osd.1 [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264]' entity='osd.1' 2024-09-10T03:37:44.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:44 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T03:37:45.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:45 smithi112 ceph-mon[27737]: purged_snaps scrub starts 2024-09-10T03:37:45.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:45 smithi112 ceph-mon[27737]: purged_snaps scrub ok 2024-09-10T03:37:45.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:45 smithi112 ceph-mon[27737]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:45.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:45 smithi112 ceph-mon[27737]: osd.1 [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264] boot 2024-09-10T03:37:45.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:45 smithi112 ceph-mon[27737]: osdmap e14: 2 total, 2 up, 2 in 2024-09-10T03:37:45.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:45 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T03:37:45.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:45 smithi169 ceph-mon[35668]: purged_snaps scrub starts 2024-09-10T03:37:45.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:45 smithi169 ceph-mon[35668]: purged_snaps scrub ok 2024-09-10T03:37:45.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:45 smithi169 ceph-mon[35668]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-10T03:37:45.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:45 smithi169 ceph-mon[35668]: osd.1 [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264] boot 2024-09-10T03:37:45.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:45 smithi169 ceph-mon[35668]: osdmap e14: 2 total, 2 up, 2 in 2024-09-10T03:37:45.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:45 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T03:37:46.063 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:37:46.090 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_2 2024-09-10T03:37:46.630 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:37:47.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.132+0000 7f6bf2b70640 1 -- 172.21.15.112:0/1415568036 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6bec101e20 msgr2=0x7f6bec102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:37:47.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.132+0000 7f6bf2b70640 1 --2- 172.21.15.112:0/1415568036 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6bec101e20 0x7f6bec102220 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6be0009880 tx=0x7f6be002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:37:47.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.132+0000 7f6bf2b70640 1 -- 172.21.15.112:0/1415568036 shutdown_connections 2024-09-10T03:37:47.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.132+0000 7f6bf2b70640 1 --2- 172.21.15.112:0/1415568036 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6bec100b60 0x7f6bec100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:37:47.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.132+0000 7f6bf2b70640 1 --2- 172.21.15.112:0/1415568036 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6bec101e20 0x7f6bec102220 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:37:47.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.132+0000 7f6bf2b70640 1 -- 172.21.15.112:0/1415568036 >> 172.21.15.112:0/1415568036 conn(0x7f6bec0fb7a0 msgr2=0x7f6bec0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:37:47.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.132+0000 7f6bf2b70640 1 -- 172.21.15.112:0/1415568036 shutdown_connections 2024-09-10T03:37:47.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.132+0000 7f6bf2b70640 1 -- 172.21.15.112:0/1415568036 wait complete. 2024-09-10T03:37:47.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.132+0000 7f6bf2b70640 1 Processor -- start 2024-09-10T03:37:47.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.132+0000 7f6bf2b70640 1 -- start start 2024-09-10T03:37:47.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.133+0000 7f6bf2b70640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6bec100b60 0x7f6bec197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:37:47.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.133+0000 7f6bf2b70640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6bec101e20 0x7f6bec197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:37:47.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.133+0000 7f6bf2b70640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec1096b0 con 0x7f6bec100b60 2024-09-10T03:37:47.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.133+0000 7f6bf2b70640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec109530 con 0x7f6bec101e20 2024-09-10T03:37:47.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.133+0000 7f6bf090c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6bec100b60 0x7f6bec197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:37:47.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.133+0000 7f6bf090c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6bec100b60 0x7f6bec197530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:42472/0 (socket says 172.21.15.112:42472) 2024-09-10T03:37:47.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.133+0000 7f6bf090c640 1 -- 172.21.15.112:0/966257654 learned_addr learned my addr 172.21.15.112:0/966257654 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:37:47.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.133+0000 7f6bebfff640 1 --2- 172.21.15.112:0/966257654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6bec101e20 0x7f6bec197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:37:47.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.133+0000 7f6bf090c640 1 -- 172.21.15.112:0/966257654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6bec101e20 msgr2=0x7f6bec197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:37:47.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.133+0000 7f6bf090c640 1 --2- 172.21.15.112:0/966257654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6bec101e20 0x7f6bec197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:37:47.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.133+0000 7f6bf090c640 1 -- 172.21.15.112:0/966257654 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bec1980f0 con 0x7f6bec100b60 2024-09-10T03:37:47.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.133+0000 7f6bebfff640 1 --2- 172.21.15.112:0/966257654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6bec101e20 0x7f6bec197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T03:37:47.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.133+0000 7f6bf090c640 1 --2- 172.21.15.112:0/966257654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6bec100b60 0x7f6bec197530 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6be0006010 tx=0x7f6be00045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:37:47.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.133+0000 7f6be9ffb640 1 -- 172.21.15.112:0/966257654 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6be003f070 con 0x7f6bec100b60 2024-09-10T03:37:47.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.133+0000 7f6bf2b70640 1 -- 172.21.15.112:0/966257654 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bec19cad0 con 0x7f6bec100b60 2024-09-10T03:37:47.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.133+0000 7f6be9ffb640 1 -- 172.21.15.112:0/966257654 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6be0004a50 con 0x7f6bec100b60 2024-09-10T03:37:47.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.133+0000 7f6be9ffb640 1 -- 172.21.15.112:0/966257654 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6be0035030 con 0x7f6bec100b60 2024-09-10T03:37:47.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.134+0000 7f6bf2b70640 1 -- 172.21.15.112:0/966257654 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bec19d060 con 0x7f6bec100b60 2024-09-10T03:37:47.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.135+0000 7f6bf2b70640 1 -- 172.21.15.112:0/966257654 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bec100fe0 con 0x7f6bec100b60 2024-09-10T03:37:47.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.135+0000 7f6be9ffb640 1 -- 172.21.15.112:0/966257654 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 18) ==== 92149+0+0 (secure 0 0 0) 0x7f6be0042070 con 0x7f6bec100b60 2024-09-10T03:37:47.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.136+0000 7f6be9ffb640 1 --2- 172.21.15.112:0/966257654 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6bdc06ddb0 0x7f6bdc070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:37:47.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.136+0000 7f6be9ffb640 1 -- 172.21.15.112:0/966257654 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2414+0+0 (secure 0 0 0) 0x7f6be00ad9b0 con 0x7f6bec100b60 2024-09-10T03:37:47.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.136+0000 7f6bebfff640 1 --2- 172.21.15.112:0/966257654 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6bdc06ddb0 0x7f6bdc070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:37:47.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.136+0000 7f6bebfff640 1 --2- 172.21.15.112:0/966257654 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6bdc06ddb0 0x7f6bdc070270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8005d00 tx=0x7f6bd8005c70 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:37:47.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.143+0000 7f6be9ffb640 1 -- 172.21.15.112:0/966257654 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6be007c220 con 0x7f6bec100b60 2024-09-10T03:37:47.328 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:37:47.326+0000 7f6bf2b70640 1 -- 172.21.15.112:0/966257654 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f6bec1001d0 con 0x7f6bdc06ddb0 2024-09-10T03:37:47.328 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:47 smithi112 ceph-mon[27737]: osdmap e15: 2 total, 2 up, 2 in 2024-09-10T03:37:47.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:47 smithi169 ceph-mon[35668]: osdmap e15: 2 total, 2 up, 2 in 2024-09-10T03:37:48.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:48 smithi112 ceph-mon[27737]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:37:48.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:48 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T03:37:48.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:48 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T03:37:48.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:48 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:48.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:48 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:48.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:48 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:48.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:48 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:37:48.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:48 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:37:48.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:48 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:48.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:48 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:48.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:48 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:37:48.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:48 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:48.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:48 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:37:48.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:48 smithi169 ceph-mon[35668]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:37:48.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:48 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T03:37:48.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:48 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T03:37:48.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:48 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:48.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:48 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:48.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:48 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:48.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:48 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:37:48.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:48 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:37:48.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:48 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:48.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:48 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:48.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:48 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:37:48.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:48 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:48.527 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:48 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:37:49.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:49 smithi169 ceph-mon[35668]: from='client.14368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:37:49.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:49 smithi169 ceph-mon[35668]: Adjusting osd_memory_target on smithi112 to 5920M 2024-09-10T03:37:49.487 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:49 smithi112 ceph-mon[27737]: from='client.14368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:37:49.488 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:49 smithi112 ceph-mon[27737]: Adjusting osd_memory_target on smithi112 to 5920M 2024-09-10T03:37:50.514 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:50 smithi112 ceph-mon[27737]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:37:50.515 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:50 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3884067466' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7d8e980-cf86-473c-805c-61ebb62e4d04"}]: dispatch 2024-09-10T03:37:50.515 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:50 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3884067466' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7d8e980-cf86-473c-805c-61ebb62e4d04"}]': finished 2024-09-10T03:37:50.515 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:50 smithi112 ceph-mon[27737]: osdmap e16: 3 total, 2 up, 3 in 2024-09-10T03:37:50.515 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:50 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T03:37:50.515 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:50 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:37:50.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:50 smithi169 ceph-mon[35668]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:37:50.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:50 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3884067466' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7d8e980-cf86-473c-805c-61ebb62e4d04"}]: dispatch 2024-09-10T03:37:50.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:50 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3884067466' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7d8e980-cf86-473c-805c-61ebb62e4d04"}]': finished 2024-09-10T03:37:50.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:50 smithi169 ceph-mon[35668]: osdmap e16: 3 total, 2 up, 3 in 2024-09-10T03:37:50.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:50 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T03:37:50.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:50 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:37:51.460 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:51 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/822788505' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T03:37:51.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:51 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/822788505' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T03:37:52.512 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:52 smithi112 ceph-mon[27737]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:37:52.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:52 smithi169 ceph-mon[35668]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:37:54.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:54 smithi169 ceph-mon[35668]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:37:54.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:54 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:54.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:54 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:54.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:54 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:54.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:54 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:37:54.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:54 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:54.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:54 smithi112 ceph-mon[27737]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:37:54.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:54 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:54.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:54 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:54.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:54 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:54.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:54 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:37:54.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:54 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:37:56.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:56 smithi112 ceph-mon[27737]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:37:56.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:56 smithi169 ceph-mon[35668]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:37:58.270 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:58 smithi112 ceph-mon[27737]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:37:58.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:58 smithi169 ceph-mon[35668]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:37:59.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:59 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-10T03:37:59.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:37:59 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:37:59.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:59 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-10T03:37:59.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:37:59 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:38:00.262 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:00 smithi112 ceph-mon[27737]: Deploying daemon osd.2 on smithi112 2024-09-10T03:38:00.263 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:00 smithi112 ceph-mon[27737]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:00.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:00 smithi169 ceph-mon[35668]: Deploying daemon osd.2 on smithi112 2024-09-10T03:38:00.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:00 smithi169 ceph-mon[35668]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:02.370 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:02 smithi112 ceph-mon[27737]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:02.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:02 smithi169 ceph-mon[35668]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:04.171 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:04 smithi112 ceph-mon[27737]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:04.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:04 smithi169 ceph-mon[35668]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:05.292 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:38:05.292 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:05.292 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:05.292 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:38:05.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:38:05.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:05.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:05.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:38:06.460 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:06 smithi112 ceph-mon[27737]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:06.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:06 smithi169 ceph-mon[35668]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:08.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:08 smithi112 ceph-mon[27737]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:08.504 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:38:08.503+0000 7f6be9ffb640 1 -- 172.21.15.112:0/966257654 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6bec1001d0 con 0x7f6bdc06ddb0 2024-09-10T03:38:08.504 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 2 on host 'smithi112' 2024-09-10T03:38:08.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:38:08.505+0000 7f6bf2b70640 1 -- 172.21.15.112:0/966257654 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6bdc06ddb0 msgr2=0x7f6bdc070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:38:08.506 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:38:08.505+0000 7f6bf2b70640 1 --2- 172.21.15.112:0/966257654 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6bdc06ddb0 0x7f6bdc070270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8005d00 tx=0x7f6bd8005c70 comp rx=0 tx=0).stop 2024-09-10T03:38:08.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:38:08.505+0000 7f6bf2b70640 1 -- 172.21.15.112:0/966257654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6bec100b60 msgr2=0x7f6bec197530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:38:08.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:38:08.505+0000 7f6bf2b70640 1 --2- 172.21.15.112:0/966257654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6bec100b60 0x7f6bec197530 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6be0006010 tx=0x7f6be00045e0 comp rx=0 tx=0).stop 2024-09-10T03:38:08.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:38:08.505+0000 7f6bf2b70640 1 -- 172.21.15.112:0/966257654 shutdown_connections 2024-09-10T03:38:08.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:38:08.505+0000 7f6bf2b70640 1 --2- 172.21.15.112:0/966257654 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6bdc06ddb0 0x7f6bdc070270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:38:08.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:38:08.505+0000 7f6bf2b70640 1 --2- 172.21.15.112:0/966257654 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6bec101e20 0x7f6bec197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:38:08.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:38:08.505+0000 7f6bf2b70640 1 --2- 172.21.15.112:0/966257654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6bec100b60 0x7f6bec197530 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:38:08.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:38:08.505+0000 7f6bf2b70640 1 -- 172.21.15.112:0/966257654 >> 172.21.15.112:0/966257654 conn(0x7f6bec0fb7a0 msgr2=0x7f6bec0ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:38:08.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:38:08.505+0000 7f6bf2b70640 1 -- 172.21.15.112:0/966257654 shutdown_connections 2024-09-10T03:38:08.507 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:38:08.505+0000 7f6bf2b70640 1 -- 172.21.15.112:0/966257654 wait complete. 2024-09-10T03:38:08.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:08 smithi169 ceph-mon[35668]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:09.696 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:09.696 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:09.696 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:09 smithi112 ceph-mon[27737]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:09.735 DEBUG:teuthology.orchestra.run.smithi112:osd.2> sudo journalctl -f -n 0 -u ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@osd.2.service 2024-09-10T03:38:09.738 INFO:tasks.cephadm:Deploying osd.3 on smithi169 with /dev/vg_nvme/lv_4... 2024-09-10T03:38:09.739 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-10T03:38:09.767 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:09.768 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:09.768 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:09 smithi169 ceph-mon[35668]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:09.996 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi169/config 2024-09-10T03:38:11.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:38:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:38:11.024+0000 7fecbc111640 -1 osd.2 0 log_to_monitors true 2024-09-10T03:38:12.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:11 smithi169 ceph-mon[35668]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:12.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:11 smithi169 ceph-mon[35668]: from='osd.2 [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-10T03:38:12.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:11 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:12.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:11 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:12.183 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:11 smithi112 ceph-mon[27737]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:12.183 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:11 smithi112 ceph-mon[27737]: from='osd.2 [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-10T03:38:12.183 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:11 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:12.183 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:11 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:12.382 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:38:12.409 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi169:vg_nvme/lv_4 2024-09-10T03:38:12.706 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi169/config 2024-09-10T03:38:12.953 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:12 smithi169 ceph-mon[35668]: from='osd.2 [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-10T03:38:12.953 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:12 smithi169 ceph-mon[35668]: osdmap e17: 3 total, 2 up, 3 in 2024-09-10T03:38:12.953 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:12 smithi169 ceph-mon[35668]: from='osd.2 [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-09-10T03:38:12.953 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:12 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T03:38:13.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.165+0000 7f21d2e12640 1 -- 172.21.15.169:0/3009129151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21cc1023d0 msgr2=0x7f21cc1047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:38:13.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.165+0000 7f21d2e12640 1 --2- 172.21.15.169:0/3009129151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21cc1023d0 0x7f21cc1047c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f21c0009880 tx=0x7f21c002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:38:13.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.166+0000 7f21d2e12640 1 -- 172.21.15.169:0/3009129151 shutdown_connections 2024-09-10T03:38:13.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.166+0000 7f21d2e12640 1 --2- 172.21.15.169:0/3009129151 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21cc104d00 0x7f21cc1070f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:38:13.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.166+0000 7f21d2e12640 1 --2- 172.21.15.169:0/3009129151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21cc1023d0 0x7f21cc1047c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:38:13.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.166+0000 7f21d2e12640 1 -- 172.21.15.169:0/3009129151 >> 172.21.15.169:0/3009129151 conn(0x7f21cc0fbff0 msgr2=0x7f21cc0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:38:13.169 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.166+0000 7f21d2e12640 1 -- 172.21.15.169:0/3009129151 shutdown_connections 2024-09-10T03:38:13.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.166+0000 7f21d2e12640 1 -- 172.21.15.169:0/3009129151 wait complete. 2024-09-10T03:38:13.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.166+0000 7f21d2e12640 1 Processor -- start 2024-09-10T03:38:13.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.166+0000 7f21d2e12640 1 -- start start 2024-09-10T03:38:13.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.166+0000 7f21d2e12640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21cc1023d0 0x7f21cc101cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:38:13.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.166+0000 7f21d2e12640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21cc104d00 0x7f21cc100380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:38:13.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.166+0000 7f21d2e12640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f21cc10a7c0 con 0x7f21cc1023d0 2024-09-10T03:38:13.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.166+0000 7f21d2e12640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f21cc10a640 con 0x7f21cc104d00 2024-09-10T03:38:13.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.166+0000 7f21cbfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21cc104d00 0x7f21cc100380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:38:13.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.167+0000 7f21cbfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21cc104d00 0x7f21cc100380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36674/0 (socket says 172.21.15.169:36674) 2024-09-10T03:38:13.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.167+0000 7f21cbfff640 1 -- 172.21.15.169:0/3431878116 learned_addr learned my addr 172.21.15.169:0/3431878116 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-09-10T03:38:13.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.167+0000 7f21d0bae640 1 --2- 172.21.15.169:0/3431878116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21cc1023d0 0x7f21cc101cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:38:13.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.167+0000 7f21cbfff640 1 -- 172.21.15.169:0/3431878116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21cc1023d0 msgr2=0x7f21cc101cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:38:13.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.167+0000 7f21cbfff640 1 --2- 172.21.15.169:0/3431878116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21cc1023d0 0x7f21cc101cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:38:13.170 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.167+0000 7f21cbfff640 1 -- 172.21.15.169:0/3431878116 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21cc1008c0 con 0x7f21cc104d00 2024-09-10T03:38:13.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.167+0000 7f21cbfff640 1 --2- 172.21.15.169:0/3431878116 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21cc104d00 0x7f21cc100380 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f21b800e970 tx=0x7f21b800ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:38:13.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.167+0000 7f21c9ffb640 1 -- 172.21.15.169:0/3431878116 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f21b800cac0 con 0x7f21cc104d00 2024-09-10T03:38:13.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.167+0000 7f21d2e12640 1 -- 172.21.15.169:0/3431878116 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21cc100bb0 con 0x7f21cc104d00 2024-09-10T03:38:13.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.167+0000 7f21c9ffb640 1 -- 172.21.15.169:0/3431878116 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f21b80054e0 con 0x7f21cc104d00 2024-09-10T03:38:13.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.167+0000 7f21c9ffb640 1 -- 172.21.15.169:0/3431878116 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f21b8010430 con 0x7f21cc104d00 2024-09-10T03:38:13.171 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.167+0000 7f21d2e12640 1 -- 172.21.15.169:0/3431878116 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21cc101140 con 0x7f21cc104d00 2024-09-10T03:38:13.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.168+0000 7f21d2e12640 1 -- 172.21.15.169:0/3431878116 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21cc1027b0 con 0x7f21cc104d00 2024-09-10T03:38:13.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.169+0000 7f21c9ffb640 1 -- 172.21.15.169:0/3431878116 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) ==== 92149+0+0 (secure 0 0 0) 0x7f21b8004ec0 con 0x7f21cc104d00 2024-09-10T03:38:13.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.169+0000 7f21c9ffb640 1 --2- 172.21.15.169:0/3431878116 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f21bc06ddb0 0x7f21bc070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:38:13.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.169+0000 7f21c9ffb640 1 -- 172.21.15.169:0/3431878116 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2566+0+0 (secure 0 0 0) 0x7f21b808d570 con 0x7f21cc104d00 2024-09-10T03:38:13.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.169+0000 7f21d0bae640 1 --2- 172.21.15.169:0/3431878116 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f21bc06ddb0 0x7f21bc070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:38:13.172 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.170+0000 7f21d0bae640 1 --2- 172.21.15.169:0/3431878116 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f21bc06ddb0 0x7f21bc070270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f21c0000c00 tx=0x7f21c0004f30 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:38:13.175 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.174+0000 7f21c9ffb640 1 -- 172.21.15.169:0/3431878116 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f21b805bd40 con 0x7f21cc104d00 2024-09-10T03:38:13.197 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:12 smithi112 ceph-mon[27737]: from='osd.2 [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-10T03:38:13.197 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:12 smithi112 ceph-mon[27737]: osdmap e17: 3 total, 2 up, 3 in 2024-09-10T03:38:13.197 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:12 smithi112 ceph-mon[27737]: from='osd.2 [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-09-10T03:38:13.197 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:12 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T03:38:13.197 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:38:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:38:12.903+0000 7fecb7871640 -1 osd.2 0 waiting for initial osdmap 2024-09-10T03:38:13.197 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:38:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:38:12.908+0000 7fecb3eab640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T03:38:13.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:13.354+0000 7f21d2e12640 1 -- 172.21.15.169:0/3431878116 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f21cc19f5a0 con 0x7f21bc06ddb0 2024-09-10T03:38:13.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:13 smithi169 ceph-mon[35668]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:13.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:13 smithi169 ceph-mon[35668]: from='osd.2 [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-09-10T03:38:13.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:13 smithi169 ceph-mon[35668]: osdmap e18: 3 total, 2 up, 3 in 2024-09-10T03:38:13.979 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:13 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T03:38:13.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:13 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T03:38:13.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:13 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T03:38:13.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:13 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T03:38:13.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:13 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:38:14.261 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:13 smithi112 ceph-mon[27737]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:14.261 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:13 smithi112 ceph-mon[27737]: from='osd.2 [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-09-10T03:38:14.261 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:13 smithi112 ceph-mon[27737]: osdmap e18: 3 total, 2 up, 3 in 2024-09-10T03:38:14.261 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:13 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T03:38:14.261 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:13 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T03:38:14.261 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:13 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T03:38:14.261 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:13 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T03:38:14.262 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:13 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:38:15.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:15 smithi169 ceph-mon[35668]: purged_snaps scrub starts 2024-09-10T03:38:15.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:15 smithi169 ceph-mon[35668]: purged_snaps scrub ok 2024-09-10T03:38:15.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:15 smithi169 ceph-mon[35668]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:38:15.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:15 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T03:38:15.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:15 smithi169 ceph-mon[35668]: osd.2 [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968] boot 2024-09-10T03:38:15.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:15 smithi169 ceph-mon[35668]: osdmap e19: 3 total, 3 up, 3 in 2024-09-10T03:38:15.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:15 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T03:38:15.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:15 smithi112 ceph-mon[27737]: purged_snaps scrub starts 2024-09-10T03:38:15.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:15 smithi112 ceph-mon[27737]: purged_snaps scrub ok 2024-09-10T03:38:15.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:15 smithi112 ceph-mon[27737]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:38:15.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:15 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T03:38:15.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:15 smithi112 ceph-mon[27737]: osd.2 [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968] boot 2024-09-10T03:38:15.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:15 smithi112 ceph-mon[27737]: osdmap e19: 3 total, 3 up, 3 in 2024-09-10T03:38:15.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:15 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T03:38:16.231 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:16.231 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-10T03:38:16.231 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: Detected new or changed devices on smithi112 2024-09-10T03:38:16.231 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-10T03:38:16.231 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:16.231 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: osdmap e20: 3 total, 3 up, 3 in 2024-09-10T03:38:16.231 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T03:38:16.231 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:16.231 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:38:16.232 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:38:16.232 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:38:16.232 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: Adjusting osd_memory_target on smithi112 to 3946M 2024-09-10T03:38:16.232 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:16.232 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:38:16.232 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:38:16.232 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:16.232 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:38:16.232 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:38:16.232 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:38:16.232 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:16.232 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/2291365336' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eec71a12-c779-49af-a994-414f77d0ec78"}]: dispatch 2024-09-10T03:38:16.232 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eec71a12-c779-49af-a994-414f77d0ec78"}]: dispatch 2024-09-10T03:38:16.233 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-10T03:38:16.233 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eec71a12-c779-49af-a994-414f77d0ec78"}]': finished 2024-09-10T03:38:16.233 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: osdmap e21: 4 total, 3 up, 4 in 2024-09-10T03:38:16.233 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:16 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T03:38:16.559 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:38:16 smithi112 sudo[68736]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T03:38:16.559 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:38:16 smithi112 sudo[68736]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T03:38:16.559 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:38:16 smithi112 sudo[68736]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T03:38:16.559 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:38:16 smithi112 sudo[68736]: pam_unix(sudo:session): session closed for user root 2024-09-10T03:38:16.559 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-09-10T03:38:16.559 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-10T03:38:16.560 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: Detected new or changed devices on smithi112 2024-09-10T03:38:16.560 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-10T03:38:16.560 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:16.560 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: osdmap e20: 3 total, 3 up, 3 in 2024-09-10T03:38:16.560 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T03:38:16.560 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:16.560 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:38:16.560 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:38:16.560 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:38:16.560 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: Adjusting osd_memory_target on smithi112 to 3946M 2024-09-10T03:38:16.560 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:16.560 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:38:16.561 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:38:16.561 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:16.561 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:38:16.561 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:38:16.561 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:38:16.561 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:16.561 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/2291365336' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eec71a12-c779-49af-a994-414f77d0ec78"}]: dispatch 2024-09-10T03:38:16.561 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eec71a12-c779-49af-a994-414f77d0ec78"}]: dispatch 2024-09-10T03:38:16.561 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-10T03:38:16.561 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eec71a12-c779-49af-a994-414f77d0ec78"}]': finished 2024-09-10T03:38:16.561 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: osdmap e21: 4 total, 3 up, 4 in 2024-09-10T03:38:16.561 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:16 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T03:38:16.828 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:38:16 smithi112 sudo[68739]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T03:38:16.828 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:38:16 smithi112 sudo[68739]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T03:38:16.828 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:38:16 smithi112 sudo[68739]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T03:38:16.829 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:38:16 smithi112 sudo[68739]: pam_unix(sudo:session): session closed for user root 2024-09-10T03:38:17.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:38:16 smithi112 sudo[68742]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T03:38:17.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:38:16 smithi112 sudo[68742]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T03:38:17.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:38:16 smithi112 sudo[68742]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T03:38:17.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:38:16 smithi112 sudo[68742]: pam_unix(sudo:session): session closed for user root 2024-09-10T03:38:17.344 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:38:17 smithi112 sudo[68748]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T03:38:17.344 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:38:17 smithi112 sudo[68748]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T03:38:17.344 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:38:17 smithi112 sudo[68748]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T03:38:17.345 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:38:17 smithi112 sudo[68748]: pam_unix(sudo:session): session closed for user root 2024-09-10T03:38:17.345 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:17 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/2011650263' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T03:38:17.345 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:17 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:17.345 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:17 smithi112 ceph-mon[27737]: osdmap e22: 4 total, 3 up, 4 in 2024-09-10T03:38:17.345 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:17 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T03:38:17.345 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:38:17 smithi112 sudo[68745]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T03:38:17.345 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:38:17 smithi112 sudo[68745]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T03:38:17.346 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:38:17 smithi112 sudo[68745]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T03:38:17.346 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:38:17 smithi112 sudo[68745]: pam_unix(sudo:session): session closed for user root 2024-09-10T03:38:17.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:17 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/2011650263' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T03:38:17.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:17 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:17.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:17 smithi169 ceph-mon[35668]: osdmap e22: 4 total, 3 up, 4 in 2024-09-10T03:38:17.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:17 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T03:38:17.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:38:17 smithi112 sudo[68751]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T03:38:17.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:38:17 smithi112 sudo[68751]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T03:38:17.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:38:17 smithi112 sudo[68751]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T03:38:17.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:38:17 smithi112 sudo[68751]: pam_unix(sudo:session): session closed for user root 2024-09-10T03:38:18.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:17 smithi112 sudo[68754]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T03:38:18.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:17 smithi112 sudo[68754]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T03:38:18.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:17 smithi112 sudo[68754]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T03:38:18.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:17 smithi112 sudo[68754]: pam_unix(sudo:session): session closed for user root 2024-09-10T03:38:18.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:18 smithi112 sudo[68757]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T03:38:18.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:18 smithi112 sudo[68757]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T03:38:18.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:18 smithi112 sudo[68757]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T03:38:18.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:18 smithi112 sudo[68757]: pam_unix(sudo:session): session closed for user root 2024-09-10T03:38:18.592 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:18 smithi169 sudo[40018]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T03:38:18.592 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:18 smithi169 sudo[40018]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T03:38:18.592 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:18 smithi169 sudo[40018]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T03:38:18.592 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:18 smithi169 sudo[40018]: pam_unix(sudo:session): session closed for user root 2024-09-10T03:38:18.599 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:18.598+0000 7f21c9ffb640 1 -- 172.21.15.169:0/3431878116 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f21b80051b0 con 0x7f21cc104d00 2024-09-10T03:38:18.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:18 smithi169 sudo[40024]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T03:38:18.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:18 smithi169 sudo[40024]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T03:38:18.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:18 smithi169 sudo[40024]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T03:38:18.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:18 smithi169 sudo[40024]: pam_unix(sudo:session): session closed for user root 2024-09-10T03:38:18.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:18 smithi169 ceph-mon[35668]: pgmap v91: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:18.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:18 smithi169 ceph-mon[35668]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T03:38:18.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:18 smithi169 ceph-mon[35668]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T03:38:18.980 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:18 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2024-09-10T03:38:18.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:18 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:38:18.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:18 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2024-09-10T03:38:18.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:18 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:38:18.981 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:18 smithi169 ceph-mon[35668]: mgrmap e19: smithi112.pqrlrb(active, since 2m), standbys: smithi169.bkwgjl 2024-09-10T03:38:19.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:18 smithi112 ceph-mon[27737]: pgmap v91: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:19.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:18 smithi112 ceph-mon[27737]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T03:38:19.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:18 smithi112 ceph-mon[27737]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T03:38:19.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:18 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2024-09-10T03:38:19.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:18 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:38:19.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:18 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2024-09-10T03:38:19.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:18 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:38:19.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:18 smithi112 ceph-mon[27737]: mgrmap e19: smithi112.pqrlrb(active, since 2m), standbys: smithi169.bkwgjl 2024-09-10T03:38:20.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:19 smithi169 ceph-mon[35668]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T03:38:20.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:19 smithi169 ceph-mon[35668]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T03:38:20.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:19 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:20.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:19 smithi169 ceph-mon[35668]: pgmap v92: 1 pgs: 1 creating+peering; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:20.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:19 smithi112 ceph-mon[27737]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T03:38:20.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:19 smithi112 ceph-mon[27737]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T03:38:20.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:19 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:20.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:19 smithi112 ceph-mon[27737]: pgmap v92: 1 pgs: 1 creating+peering; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:21.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:20 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:38:21.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:20 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:38:21.747 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:21 smithi169 ceph-mon[35668]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:22.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:21 smithi112 ceph-mon[27737]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:24.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:23 smithi169 ceph-mon[35668]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:24.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:23 smithi112 ceph-mon[27737]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:25.137 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:24 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-10T03:38:25.137 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:24 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:38:25.237 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:24 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-10T03:38:25.237 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:24 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:38:26.177 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:25 smithi169 ceph-mon[35668]: Deploying daemon osd.3 on smithi169 2024-09-10T03:38:26.177 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:25 smithi169 ceph-mon[35668]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:26.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:25 smithi112 ceph-mon[27737]: Deploying daemon osd.3 on smithi169 2024-09-10T03:38:26.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:25 smithi112 ceph-mon[27737]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:27.941 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:27 smithi169 ceph-mon[35668]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:28.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:27 smithi112 ceph-mon[27737]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:30.080 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:29 smithi169 ceph-mon[35668]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:30.310 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:29 smithi112 ceph-mon[27737]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:31.217 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:30 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:38:31.217 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:30 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:31.217 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:30 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:31.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:30 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:38:31.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:30 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:31.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:30 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:32.100 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:31 smithi169 ceph-mon[35668]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:32.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:31 smithi112 ceph-mon[27737]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:34.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:33 smithi169 ceph-mon[35668]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:34.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:33 smithi112 ceph-mon[27737]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:35.346 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:35.346 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:35.346 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:38:35.347 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:38:35.347 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:35.347 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:35 smithi169 ceph-mon[35668]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:35.347 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:38:35.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:35.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:35.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:38:35.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:38:35.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:35.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:35 smithi112 ceph-mon[27737]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:35.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:38:35.685 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:35.684+0000 7f21c9ffb640 1 -- 172.21.15.169:0/3431878116 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f21cc19f5a0 con 0x7f21bc06ddb0 2024-09-10T03:38:35.685 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 3 on host 'smithi169' 2024-09-10T03:38:35.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:35.687+0000 7f21d2e12640 1 -- 172.21.15.169:0/3431878116 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f21bc06ddb0 msgr2=0x7f21bc070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:38:35.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:35.687+0000 7f21d2e12640 1 --2- 172.21.15.169:0/3431878116 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f21bc06ddb0 0x7f21bc070270 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f21c0000c00 tx=0x7f21c0004f30 comp rx=0 tx=0).stop 2024-09-10T03:38:35.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:35.687+0000 7f21d2e12640 1 -- 172.21.15.169:0/3431878116 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21cc104d00 msgr2=0x7f21cc100380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:38:35.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:35.687+0000 7f21d2e12640 1 --2- 172.21.15.169:0/3431878116 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21cc104d00 0x7f21cc100380 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f21b800e970 tx=0x7f21b800ee30 comp rx=0 tx=0).stop 2024-09-10T03:38:35.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:35.687+0000 7f21d2e12640 1 -- 172.21.15.169:0/3431878116 shutdown_connections 2024-09-10T03:38:35.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:35.687+0000 7f21d2e12640 1 --2- 172.21.15.169:0/3431878116 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f21bc06ddb0 0x7f21bc070270 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:38:35.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:35.687+0000 7f21d2e12640 1 --2- 172.21.15.169:0/3431878116 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f21cc104d00 0x7f21cc100380 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:38:35.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:35.687+0000 7f21d2e12640 1 --2- 172.21.15.169:0/3431878116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21cc1023d0 0x7f21cc101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:38:35.688 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:35.687+0000 7f21d2e12640 1 -- 172.21.15.169:0/3431878116 >> 172.21.15.169:0/3431878116 conn(0x7f21cc0fbff0 msgr2=0x7f21cc0fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:38:35.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:35.687+0000 7f21d2e12640 1 -- 172.21.15.169:0/3431878116 shutdown_connections 2024-09-10T03:38:35.689 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:35.687+0000 7f21d2e12640 1 -- 172.21.15.169:0/3431878116 wait complete. 2024-09-10T03:38:36.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:36 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:38:36.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:36 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:36.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:36 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:36.336 DEBUG:teuthology.orchestra.run.smithi169:osd.3> sudo journalctl -f -n 0 -u ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@osd.3.service 2024-09-10T03:38:36.339 INFO:tasks.cephadm:Deploying osd.4 on smithi169 with /dev/vg_nvme/lv_3... 2024-09-10T03:38:36.339 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-10T03:38:36.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:36 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:38:36.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:36 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:36.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:36 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:37.026 INFO:journalctl@ceph.osd.3.smithi169.stdout:Sep 10 03:38:36 smithi169 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-3[41617]: 2024-09-10T03:38:36.683+0000 7f544b505640 -1 osd.3 0 log_to_monitors true 2024-09-10T03:38:37.054 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi169/config 2024-09-10T03:38:37.441 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:37 smithi169 ceph-mon[35668]: from='osd.3 [v2:172.21.15.169:6800/3830555794,v1:172.21.15.169:6801/3830555794]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T03:38:37.441 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:37 smithi169 ceph-mon[35668]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T03:38:37.441 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:37 smithi169 ceph-mon[35668]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:37.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:37 smithi112 ceph-mon[27737]: from='osd.3 [v2:172.21.15.169:6800/3830555794,v1:172.21.15.169:6801/3830555794]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T03:38:37.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:37 smithi112 ceph-mon[27737]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T03:38:37.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:37 smithi112 ceph-mon[27737]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:38.469 INFO:journalctl@ceph.osd.3.smithi169.stdout:Sep 10 03:38:38 smithi169 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-3[41617]: 2024-09-10T03:38:38.222+0000 7f5446c65640 -1 osd.3 0 waiting for initial osdmap 2024-09-10T03:38:38.469 INFO:journalctl@ceph.osd.3.smithi169.stdout:Sep 10 03:38:38 smithi169 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-3[41617]: 2024-09-10T03:38:38.228+0000 7f544329f640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T03:38:38.469 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:38 smithi169 ceph-mon[35668]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-10T03:38:38.469 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:38 smithi169 ceph-mon[35668]: from='osd.3 [v2:172.21.15.169:6800/3830555794,v1:172.21.15.169:6801/3830555794]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-09-10T03:38:38.469 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:38 smithi169 ceph-mon[35668]: osdmap e23: 4 total, 3 up, 4 in 2024-09-10T03:38:38.469 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:38 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T03:38:38.469 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:38 smithi169 ceph-mon[35668]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-09-10T03:38:38.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:38 smithi112 ceph-mon[27737]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-10T03:38:38.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:38 smithi112 ceph-mon[27737]: from='osd.3 [v2:172.21.15.169:6800/3830555794,v1:172.21.15.169:6801/3830555794]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-09-10T03:38:38.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:38 smithi112 ceph-mon[27737]: osdmap e23: 4 total, 3 up, 4 in 2024-09-10T03:38:38.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:38 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T03:38:38.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:38 smithi112 ceph-mon[27737]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-09-10T03:38:39.488 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:39 smithi169 ceph-mon[35668]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-09-10T03:38:39.489 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:39 smithi169 ceph-mon[35668]: osdmap e24: 4 total, 3 up, 4 in 2024-09-10T03:38:39.489 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:39 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T03:38:39.489 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:39 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T03:38:39.489 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:39 smithi169 ceph-mon[35668]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:39.489 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:39 smithi169 ceph-mon[35668]: osd.3 [v2:172.21.15.169:6800/3830555794,v1:172.21.15.169:6801/3830555794] boot 2024-09-10T03:38:39.489 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:39 smithi169 ceph-mon[35668]: osdmap e25: 4 total, 4 up, 4 in 2024-09-10T03:38:39.489 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:39 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T03:38:39.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:39 smithi112 ceph-mon[27737]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-09-10T03:38:39.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:39 smithi112 ceph-mon[27737]: osdmap e24: 4 total, 3 up, 4 in 2024-09-10T03:38:39.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:39 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T03:38:39.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:39 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T03:38:39.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:39 smithi112 ceph-mon[27737]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-09-10T03:38:39.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:39 smithi112 ceph-mon[27737]: osd.3 [v2:172.21.15.169:6800/3830555794,v1:172.21.15.169:6801/3830555794] boot 2024-09-10T03:38:39.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:39 smithi112 ceph-mon[27737]: osdmap e25: 4 total, 4 up, 4 in 2024-09-10T03:38:39.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:39 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T03:38:40.497 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:40 smithi169 ceph-mon[35668]: purged_snaps scrub starts 2024-09-10T03:38:40.498 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:40 smithi169 ceph-mon[35668]: purged_snaps scrub ok 2024-09-10T03:38:40.498 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:40 smithi169 ceph-mon[35668]: Detected new or changed devices on smithi169 2024-09-10T03:38:40.498 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:40 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:40.498 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:40 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:40.498 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:40 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:38:40.498 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:40 smithi169 ceph-mon[35668]: Adjusting osd_memory_target on smithi169 to 14912M 2024-09-10T03:38:40.498 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:40 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:40.498 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:40 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:38:40.498 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:40 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:38:40.498 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:40 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:40.499 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:40 smithi169 ceph-mon[35668]: osdmap e26: 4 total, 4 up, 4 in 2024-09-10T03:38:40.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:40 smithi112 ceph-mon[27737]: purged_snaps scrub starts 2024-09-10T03:38:40.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:40 smithi112 ceph-mon[27737]: purged_snaps scrub ok 2024-09-10T03:38:40.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:40 smithi112 ceph-mon[27737]: Detected new or changed devices on smithi169 2024-09-10T03:38:40.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:40.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:40.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:38:40.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:40 smithi112 ceph-mon[27737]: Adjusting osd_memory_target on smithi169 to 14912M 2024-09-10T03:38:40.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:40.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:38:40.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:38:40.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:40 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:38:40.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:40 smithi112 ceph-mon[27737]: osdmap e26: 4 total, 4 up, 4 in 2024-09-10T03:38:41.124 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:38:41.150 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi169:vg_nvme/lv_3 2024-09-10T03:38:41.399 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi169/config 2024-09-10T03:38:41.520 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:41 smithi169 ceph-mon[35668]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-10T03:38:41.520 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:41 smithi169 ceph-mon[35668]: osdmap e27: 4 total, 4 up, 4 in 2024-09-10T03:38:41.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:41 smithi112 ceph-mon[27737]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-10T03:38:41.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:41 smithi112 ceph-mon[27737]: osdmap e27: 4 total, 4 up, 4 in 2024-09-10T03:38:41.891 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.856+0000 7ff3768fe640 1 -- 172.21.15.169:0/3194094236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff370102f60 msgr2=0x7ff370103360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:38:41.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.856+0000 7ff3768fe640 1 --2- 172.21.15.169:0/3194094236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff370102f60 0x7ff370103360 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff360009880 tx=0x7ff36002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:38:41.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.856+0000 7ff3768fe640 1 -- 172.21.15.169:0/3194094236 shutdown_connections 2024-09-10T03:38:41.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.856+0000 7ff3768fe640 1 --2- 172.21.15.169:0/3194094236 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff370103930 0x7ff3700ffa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:38:41.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.856+0000 7ff3768fe640 1 --2- 172.21.15.169:0/3194094236 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff370102f60 0x7ff370103360 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:38:41.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.856+0000 7ff3768fe640 1 -- 172.21.15.169:0/3194094236 >> 172.21.15.169:0/3194094236 conn(0x7ff3700fb790 msgr2=0x7ff3700fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:38:41.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.856+0000 7ff3768fe640 1 -- 172.21.15.169:0/3194094236 shutdown_connections 2024-09-10T03:38:41.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.856+0000 7ff3768fe640 1 -- 172.21.15.169:0/3194094236 wait complete. 2024-09-10T03:38:41.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.857+0000 7ff3768fe640 1 Processor -- start 2024-09-10T03:38:41.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.857+0000 7ff3768fe640 1 -- start start 2024-09-10T03:38:41.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.857+0000 7ff3768fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff370102f60 0x7ff370197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:38:41.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.857+0000 7ff3768fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff370103930 0x7ff370197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:38:41.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.857+0000 7ff3768fe640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff37010a850 con 0x7ff370102f60 2024-09-10T03:38:41.893 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.857+0000 7ff3768fe640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff37010a6d0 con 0x7ff370103930 2024-09-10T03:38:41.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.857+0000 7ff367fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff370103930 0x7ff370197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:38:41.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.857+0000 7ff367fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff370103930 0x7ff370197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:46336/0 (socket says 172.21.15.169:46336) 2024-09-10T03:38:41.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.857+0000 7ff367fff640 1 -- 172.21.15.169:0/2786024631 learned_addr learned my addr 172.21.15.169:0/2786024631 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-09-10T03:38:41.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.857+0000 7ff36ffff640 1 --2- 172.21.15.169:0/2786024631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff370102f60 0x7ff370197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:38:41.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.857+0000 7ff367fff640 1 -- 172.21.15.169:0/2786024631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff370102f60 msgr2=0x7ff370197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:38:41.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.857+0000 7ff367fff640 1 --2- 172.21.15.169:0/2786024631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff370102f60 0x7ff370197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:38:41.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.857+0000 7ff367fff640 1 -- 172.21.15.169:0/2786024631 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff370198150 con 0x7ff370103930 2024-09-10T03:38:41.894 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.857+0000 7ff36ffff640 1 --2- 172.21.15.169:0/2786024631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff370102f60 0x7ff370197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T03:38:41.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.857+0000 7ff367fff640 1 --2- 172.21.15.169:0/2786024631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff370103930 0x7ff370197ad0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff35800b3d0 tx=0x7ff35800b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:38:41.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.857+0000 7ff36dffb640 1 -- 172.21.15.169:0/2786024631 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff358015020 con 0x7ff370103930 2024-09-10T03:38:41.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.858+0000 7ff36dffb640 1 -- 172.21.15.169:0/2786024631 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff358011af0 con 0x7ff370103930 2024-09-10T03:38:41.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.858+0000 7ff3768fe640 1 -- 172.21.15.169:0/2786024631 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff37019cb90 con 0x7ff370103930 2024-09-10T03:38:41.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.858+0000 7ff36dffb640 1 -- 172.21.15.169:0/2786024631 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff358010d30 con 0x7ff370103930 2024-09-10T03:38:41.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.858+0000 7ff3768fe640 1 -- 172.21.15.169:0/2786024631 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff37019d1a0 con 0x7ff370103930 2024-09-10T03:38:41.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.859+0000 7ff3768fe640 1 -- 172.21.15.169:0/2786024631 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff370103360 con 0x7ff370103930 2024-09-10T03:38:41.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.859+0000 7ff36dffb640 1 -- 172.21.15.169:0/2786024631 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7ff358018020 con 0x7ff370103930 2024-09-10T03:38:41.895 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.860+0000 7ff36dffb640 1 --2- 172.21.15.169:0/2786024631 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff35c06de30 0x7ff35c0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:38:41.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.860+0000 7ff36dffb640 1 -- 172.21.15.169:0/2786024631 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3774+0+0 (secure 0 0 0) 0x7ff35808cfc0 con 0x7ff370103930 2024-09-10T03:38:41.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.860+0000 7ff36ffff640 1 --2- 172.21.15.169:0/2786024631 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff35c06de30 0x7ff35c0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:38:41.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.860+0000 7ff36ffff640 1 --2- 172.21.15.169:0/2786024631 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff35c06de30 0x7ff35c0702f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff360000c00 tx=0x7ff360005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:38:41.896 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:41.864+0000 7ff36dffb640 1 -- 172.21.15.169:0/2786024631 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff35801f020 con 0x7ff370103930 2024-09-10T03:38:42.043 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:38:42.042+0000 7ff3768fe640 1 -- 172.21.15.169:0/2786024631 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ff370062630 con 0x7ff35c06de30 2024-09-10T03:38:43.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:42 smithi169 ceph-mon[35668]: osdmap e28: 4 total, 4 up, 4 in 2024-09-10T03:38:43.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:42 smithi169 ceph-mon[35668]: from='client.24183 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:38:43.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:42 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T03:38:43.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:42 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T03:38:43.027 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:42 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:38:43.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:42 smithi112 ceph-mon[27737]: osdmap e28: 4 total, 4 up, 4 in 2024-09-10T03:38:43.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:42 smithi112 ceph-mon[27737]: from='client.24183 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:38:43.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:42 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T03:38:43.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:42 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T03:38:43.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:42 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:38:43.982 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:43 smithi169 ceph-mon[35668]: osdmap e29: 4 total, 4 up, 4 in 2024-09-10T03:38:43.982 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:43 smithi169 ceph-mon[35668]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-10T03:38:44.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:43 smithi112 ceph-mon[27737]: osdmap e29: 4 total, 4 up, 4 in 2024-09-10T03:38:44.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:43 smithi112 ceph-mon[27737]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-10T03:38:44.918 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:44 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/1046111863' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77c831ef-a06d-4f4f-ad2a-c2cb04613fe0"}]: dispatch 2024-09-10T03:38:44.918 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:44 smithi169 ceph-mon[35668]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77c831ef-a06d-4f4f-ad2a-c2cb04613fe0"}]: dispatch 2024-09-10T03:38:44.918 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:44 smithi169 ceph-mon[35668]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77c831ef-a06d-4f4f-ad2a-c2cb04613fe0"}]': finished 2024-09-10T03:38:44.918 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:44 smithi169 ceph-mon[35668]: osdmap e30: 5 total, 4 up, 5 in 2024-09-10T03:38:44.919 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:44 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T03:38:45.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:44 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/1046111863' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77c831ef-a06d-4f4f-ad2a-c2cb04613fe0"}]: dispatch 2024-09-10T03:38:45.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:44 smithi112 ceph-mon[27737]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77c831ef-a06d-4f4f-ad2a-c2cb04613fe0"}]: dispatch 2024-09-10T03:38:45.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:44 smithi112 ceph-mon[27737]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77c831ef-a06d-4f4f-ad2a-c2cb04613fe0"}]': finished 2024-09-10T03:38:45.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:44 smithi112 ceph-mon[27737]: osdmap e30: 5 total, 4 up, 5 in 2024-09-10T03:38:45.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:44 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T03:38:45.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:45 smithi112 ceph-mon[27737]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail 2024-09-10T03:38:45.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:45 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/4056293727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T03:38:46.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:45 smithi169 ceph-mon[35668]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 117 MiB used, 357 GiB / 358 GiB avail 2024-09-10T03:38:46.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:45 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/4056293727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T03:38:48.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:47 smithi169 ceph-mon[35668]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-10T03:38:48.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:47 smithi112 ceph-mon[27737]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-10T03:38:50.117 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:49 smithi169 ceph-mon[35668]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-10T03:38:50.117 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:49 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:38:50.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:49 smithi112 ceph-mon[27737]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-10T03:38:50.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:49 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:38:52.150 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:51 smithi169 ceph-mon[35668]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-10T03:38:52.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:51 smithi112 ceph-mon[27737]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-09-10T03:38:53.983 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:53 smithi169 ceph-mon[35668]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T03:38:53.983 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:53 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-10T03:38:53.983 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:53 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:38:53.983 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:53 smithi169 ceph-mon[35668]: Deploying daemon osd.4 on smithi169 2024-09-10T03:38:54.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:53 smithi112 ceph-mon[27737]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T03:38:54.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:53 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-10T03:38:54.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:53 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:38:54.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:53 smithi112 ceph-mon[27737]: Deploying daemon osd.4 on smithi169 2024-09-10T03:38:56.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:55 smithi169 ceph-mon[35668]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-10T03:38:56.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:55 smithi112 ceph-mon[27737]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-09-10T03:38:57.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:57 smithi169 ceph-mon[35668]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-10T03:38:58.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:57 smithi112 ceph-mon[27737]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-10T03:39:00.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:59 smithi169 ceph-mon[35668]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-10T03:39:00.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:59 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:39:00.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:59 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:00.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:38:59 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:00.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:59 smithi112 ceph-mon[27737]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-10T03:39:00.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:59 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:39:00.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:59 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:00.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:38:59 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:02.234 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:01 smithi169 ceph-mon[35668]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-10T03:39:02.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:01 smithi112 ceph-mon[27737]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-10T03:39:03.097 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:03.095+0000 7ff36dffb640 1 -- 172.21.15.169:0/2786024631 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff370062630 con 0x7ff35c06de30 2024-09-10T03:39:03.097 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 4 on host 'smithi169' 2024-09-10T03:39:03.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:03.097+0000 7ff3768fe640 1 -- 172.21.15.169:0/2786024631 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff35c06de30 msgr2=0x7ff35c0702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:03.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:03.097+0000 7ff3768fe640 1 --2- 172.21.15.169:0/2786024631 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff35c06de30 0x7ff35c0702f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff360000c00 tx=0x7ff360005a20 comp rx=0 tx=0).stop 2024-09-10T03:39:03.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:03.097+0000 7ff3768fe640 1 -- 172.21.15.169:0/2786024631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff370103930 msgr2=0x7ff370197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:03.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:03.097+0000 7ff3768fe640 1 --2- 172.21.15.169:0/2786024631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff370103930 0x7ff370197ad0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff35800b3d0 tx=0x7ff35800b890 comp rx=0 tx=0).stop 2024-09-10T03:39:03.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:03.097+0000 7ff3768fe640 1 -- 172.21.15.169:0/2786024631 shutdown_connections 2024-09-10T03:39:03.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:03.097+0000 7ff3768fe640 1 --2- 172.21.15.169:0/2786024631 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff35c06de30 0x7ff35c0702f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:03.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:03.097+0000 7ff3768fe640 1 --2- 172.21.15.169:0/2786024631 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff370103930 0x7ff370197ad0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:03.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:03.097+0000 7ff3768fe640 1 --2- 172.21.15.169:0/2786024631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff370102f60 0x7ff370197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:03.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:03.097+0000 7ff3768fe640 1 -- 172.21.15.169:0/2786024631 >> 172.21.15.169:0/2786024631 conn(0x7ff3700fb790 msgr2=0x7ff370106230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:03.099 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:03.097+0000 7ff3768fe640 1 -- 172.21.15.169:0/2786024631 shutdown_connections 2024-09-10T03:39:03.100 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:03.097+0000 7ff3768fe640 1 -- 172.21.15.169:0/2786024631 wait complete. 2024-09-10T03:39:03.712 DEBUG:teuthology.orchestra.run.smithi169:osd.4> sudo journalctl -f -n 0 -u ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@osd.4.service 2024-09-10T03:39:03.717 INFO:tasks.cephadm:Deploying osd.5 on smithi169 with /dev/vg_nvme/lv_2... 2024-09-10T03:39:03.717 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-10T03:39:04.028 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi169/config 2024-09-10T03:39:04.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:04 smithi169 ceph-mon[35668]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-10T03:39:04.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:04 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:04.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:04 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:04.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:04 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:04.272 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:04 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:04.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:04 smithi112 ceph-mon[27737]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-10T03:39:04.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:04 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:04.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:04 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:04.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:04 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:04.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:04 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:05.376 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:39:05.376 INFO:journalctl@ceph.osd.4.smithi169.stdout:Sep 10 03:39:05 smithi169 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-4[48328]: 2024-09-10T03:39:05.197+0000 7f04e59e8640 -1 osd.4 0 log_to_monitors true 2024-09-10T03:39:05.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:39:06.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:06 smithi169 ceph-mon[35668]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-10T03:39:06.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:06 smithi169 ceph-mon[35668]: from='osd.4 [v2:172.21.15.169:6808/2285032153,v1:172.21.15.169:6809/2285032153]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T03:39:06.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:06 smithi169 ceph-mon[35668]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T03:39:06.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:06 smithi112 ceph-mon[27737]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-10T03:39:06.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:06 smithi112 ceph-mon[27737]: from='osd.4 [v2:172.21.15.169:6808/2285032153,v1:172.21.15.169:6809/2285032153]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T03:39:06.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:06 smithi112 ceph-mon[27737]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T03:39:07.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:07 smithi169 ceph-mon[35668]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-10T03:39:07.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:07 smithi169 ceph-mon[35668]: osdmap e31: 5 total, 4 up, 5 in 2024-09-10T03:39:07.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:07 smithi169 ceph-mon[35668]: from='osd.4 [v2:172.21.15.169:6808/2285032153,v1:172.21.15.169:6809/2285032153]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-09-10T03:39:07.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:07 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T03:39:07.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:07 smithi169 ceph-mon[35668]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-09-10T03:39:07.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:07 smithi169 ceph-mon[35668]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-09-10T03:39:07.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:07 smithi169 ceph-mon[35668]: osdmap e32: 5 total, 4 up, 5 in 2024-09-10T03:39:07.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:07 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T03:39:07.333 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-10T03:39:07.359 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi169:vg_nvme/lv_2 2024-09-10T03:39:07.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:07 smithi112 ceph-mon[27737]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-10T03:39:07.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:07 smithi112 ceph-mon[27737]: osdmap e31: 5 total, 4 up, 5 in 2024-09-10T03:39:07.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:07 smithi112 ceph-mon[27737]: from='osd.4 [v2:172.21.15.169:6808/2285032153,v1:172.21.15.169:6809/2285032153]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-09-10T03:39:07.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:07 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T03:39:07.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:07 smithi112 ceph-mon[27737]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-09-10T03:39:07.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:07 smithi112 ceph-mon[27737]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-09-10T03:39:07.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:07 smithi112 ceph-mon[27737]: osdmap e32: 5 total, 4 up, 5 in 2024-09-10T03:39:07.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:07 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T03:39:07.950 INFO:journalctl@ceph.osd.4.smithi169.stdout:Sep 10 03:39:07 smithi169 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-4[48328]: 2024-09-10T03:39:07.912+0000 7f04e1148640 -1 osd.4 0 waiting for initial osdmap 2024-09-10T03:39:07.951 INFO:journalctl@ceph.osd.4.smithi169.stdout:Sep 10 03:39:07 smithi169 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-4[48328]: 2024-09-10T03:39:07.922+0000 7f04d977a640 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T03:39:08.003 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi169/config 2024-09-10T03:39:08.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:08 smithi169 ceph-mon[35668]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-10T03:39:08.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:08 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T03:39:08.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:08 smithi169 ceph-mon[35668]: from='osd.4 ' entity='osd.4' 2024-09-10T03:39:08.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:08 smithi169 ceph-mon[35668]: osd.4 [v2:172.21.15.169:6808/2285032153,v1:172.21.15.169:6809/2285032153] boot 2024-09-10T03:39:08.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:08 smithi169 ceph-mon[35668]: osdmap e33: 5 total, 5 up, 5 in 2024-09-10T03:39:08.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:08 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T03:39:08.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.508+0000 7f8a6fde2640 1 -- 172.21.15.169:0/2219656624 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a68101a80 msgr2=0x7f8a68101f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:08.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.508+0000 7f8a6fde2640 1 --2- 172.21.15.169:0/2219656624 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a68101a80 0x7f8a68101f00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8a58009880 tx=0x7f8a5802a4a0 comp rx=0 tx=0).stop 2024-09-10T03:39:08.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.508+0000 7f8a6fde2640 1 -- 172.21.15.169:0/2219656624 shutdown_connections 2024-09-10T03:39:08.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.508+0000 7f8a6fde2640 1 --2- 172.21.15.169:0/2219656624 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a68101a80 0x7f8a68101f00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:08.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.508+0000 7f8a6fde2640 1 --2- 172.21.15.169:0/2219656624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a68100880 0x7f8a68100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:08.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.508+0000 7f8a6fde2640 1 -- 172.21.15.169:0/2219656624 >> 172.21.15.169:0/2219656624 conn(0x7f8a680fc010 msgr2=0x7f8a680fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:08.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.508+0000 7f8a6fde2640 1 -- 172.21.15.169:0/2219656624 shutdown_connections 2024-09-10T03:39:08.509 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.508+0000 7f8a6fde2640 1 -- 172.21.15.169:0/2219656624 wait complete. 2024-09-10T03:39:08.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.509+0000 7f8a6fde2640 1 Processor -- start 2024-09-10T03:39:08.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.509+0000 7f8a6fde2640 1 -- start start 2024-09-10T03:39:08.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.509+0000 7f8a6fde2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a68100880 0x7f8a68199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:08.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.509+0000 7f8a6fde2640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a68101a80 0x7f8a68199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:08.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.509+0000 7f8a6fde2640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8a68109740 con 0x7f8a68101a80 2024-09-10T03:39:08.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.509+0000 7f8a6fde2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8a681095c0 con 0x7f8a68100880 2024-09-10T03:39:08.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.509+0000 7f8a6db7e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a68100880 0x7f8a68199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:08.511 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.509+0000 7f8a6db7e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a68100880 0x7f8a68199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:42202/0 (socket says 172.21.15.169:42202) 2024-09-10T03:39:08.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.509+0000 7f8a6db7e640 1 -- 172.21.15.169:0/490058304 learned_addr learned my addr 172.21.15.169:0/490058304 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-09-10T03:39:08.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.509+0000 7f8a6d37d640 1 --2- 172.21.15.169:0/490058304 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a68101a80 0x7f8a68199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:08.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.510+0000 7f8a6db7e640 1 -- 172.21.15.169:0/490058304 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a68101a80 msgr2=0x7f8a68199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:08.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.510+0000 7f8a6db7e640 1 --2- 172.21.15.169:0/490058304 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a68101a80 0x7f8a68199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:08.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.510+0000 7f8a6db7e640 1 -- 172.21.15.169:0/490058304 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6819a320 con 0x7f8a68100880 2024-09-10T03:39:08.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.510+0000 7f8a6db7e640 1 --2- 172.21.15.169:0/490058304 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a68100880 0x7f8a68199760 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400b3d0 tx=0x7f8a6400b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:08.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.510+0000 7f8a5effd640 1 -- 172.21.15.169:0/490058304 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8a64015020 con 0x7f8a68100880 2024-09-10T03:39:08.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.510+0000 7f8a5effd640 1 -- 172.21.15.169:0/490058304 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8a64011af0 con 0x7f8a68100880 2024-09-10T03:39:08.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.510+0000 7f8a6fde2640 1 -- 172.21.15.169:0/490058304 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a6819ed60 con 0x7f8a68100880 2024-09-10T03:39:08.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.510+0000 7f8a5effd640 1 -- 172.21.15.169:0/490058304 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8a64010da0 con 0x7f8a68100880 2024-09-10T03:39:08.512 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.510+0000 7f8a6fde2640 1 -- 172.21.15.169:0/490058304 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a6819f280 con 0x7f8a68100880 2024-09-10T03:39:08.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.511+0000 7f8a6fde2640 1 -- 172.21.15.169:0/490058304 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a68100c80 con 0x7f8a68100880 2024-09-10T03:39:08.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.512+0000 7f8a5effd640 1 -- 172.21.15.169:0/490058304 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f8a64018020 con 0x7f8a68100880 2024-09-10T03:39:08.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.512+0000 7f8a5effd640 1 --2- 172.21.15.169:0/490058304 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f8a5406de30 0x7f8a540702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:08.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.512+0000 7f8a5effd640 1 -- 172.21.15.169:0/490058304 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4206+0+0 (secure 0 0 0) 0x7f8a6408d1c0 con 0x7f8a68100880 2024-09-10T03:39:08.513 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.512+0000 7f8a6d37d640 1 --2- 172.21.15.169:0/490058304 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f8a5406de30 0x7f8a540702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:08.514 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.513+0000 7f8a6d37d640 1 --2- 172.21.15.169:0/490058304 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f8a5406de30 0x7f8a540702f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8a58000c00 tx=0x7f8a580058d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:08.517 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.517+0000 7f8a5effd640 1 -- 172.21.15.169:0/490058304 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8a6401f020 con 0x7f8a68100880 2024-09-10T03:39:08.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:08 smithi112 ceph-mon[27737]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-09-10T03:39:08.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:08 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T03:39:08.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:08 smithi112 ceph-mon[27737]: from='osd.4 ' entity='osd.4' 2024-09-10T03:39:08.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:08 smithi112 ceph-mon[27737]: osd.4 [v2:172.21.15.169:6808/2285032153,v1:172.21.15.169:6809/2285032153] boot 2024-09-10T03:39:08.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:08 smithi112 ceph-mon[27737]: osdmap e33: 5 total, 5 up, 5 in 2024-09-10T03:39:08.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:08 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T03:39:08.697 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:08.696+0000 7f8a6fde2640 1 -- 172.21.15.169:0/490058304 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f8a6806aab0 con 0x7f8a5406de30 2024-09-10T03:39:09.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:09 smithi169 ceph-mon[35668]: purged_snaps scrub starts 2024-09-10T03:39:09.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:09 smithi169 ceph-mon[35668]: purged_snaps scrub ok 2024-09-10T03:39:09.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T03:39:09.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T03:39:09.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:39:09.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:09.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:09.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:39:09.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:39:09.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:09 smithi112 ceph-mon[27737]: purged_snaps scrub starts 2024-09-10T03:39:09.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:09 smithi112 ceph-mon[27737]: purged_snaps scrub ok 2024-09-10T03:39:09.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T03:39:09.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T03:39:09.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:39:09.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:09.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:09.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:39:09.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:39:10.489 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:10 smithi169 ceph-mon[35668]: from='client.24201 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:39:10.489 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:10 smithi169 ceph-mon[35668]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:10.489 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:10 smithi169 ceph-mon[35668]: Detected new or changed devices on smithi169 2024-09-10T03:39:10.489 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:10 smithi169 ceph-mon[35668]: Adjusting osd_memory_target on smithi169 to 7456M 2024-09-10T03:39:10.489 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:10 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:10.489 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:10 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:39:10.489 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:10 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:39:10.489 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:10 smithi169 ceph-mon[35668]: osdmap e34: 5 total, 5 up, 5 in 2024-09-10T03:39:10.489 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:10 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:10.489 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:10 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:39:10.490 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:10 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:39:10.490 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:10 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:39:10.490 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:10 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:10.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:10 smithi112 ceph-mon[27737]: from='client.24201 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:39:10.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:10 smithi112 ceph-mon[27737]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:10.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:10 smithi112 ceph-mon[27737]: Detected new or changed devices on smithi169 2024-09-10T03:39:10.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:10 smithi112 ceph-mon[27737]: Adjusting osd_memory_target on smithi169 to 7456M 2024-09-10T03:39:10.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:10 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:10.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:10 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:39:10.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:10 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:39:10.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:10 smithi112 ceph-mon[27737]: osdmap e34: 5 total, 5 up, 5 in 2024-09-10T03:39:10.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:10 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:10.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:10 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:39:10.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:10 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:39:10.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:10 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:39:10.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:10 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:12.424 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:12 smithi169 ceph-mon[35668]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:12.424 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:12 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/2306402886' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "346b5969-a50d-4340-a839-ef1085e248fd"}]: dispatch 2024-09-10T03:39:12.424 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:12 smithi169 ceph-mon[35668]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "346b5969-a50d-4340-a839-ef1085e248fd"}]: dispatch 2024-09-10T03:39:12.424 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:12 smithi169 ceph-mon[35668]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "346b5969-a50d-4340-a839-ef1085e248fd"}]': finished 2024-09-10T03:39:12.424 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:12 smithi169 ceph-mon[35668]: osdmap e35: 6 total, 5 up, 6 in 2024-09-10T03:39:12.424 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:12 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T03:39:12.424 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:12 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/2189767594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T03:39:12.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:12 smithi112 ceph-mon[27737]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:12.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:12 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/2306402886' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "346b5969-a50d-4340-a839-ef1085e248fd"}]: dispatch 2024-09-10T03:39:12.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:12 smithi112 ceph-mon[27737]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "346b5969-a50d-4340-a839-ef1085e248fd"}]: dispatch 2024-09-10T03:39:12.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:12 smithi112 ceph-mon[27737]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "346b5969-a50d-4340-a839-ef1085e248fd"}]': finished 2024-09-10T03:39:12.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:12 smithi112 ceph-mon[27737]: osdmap e35: 6 total, 5 up, 6 in 2024-09-10T03:39:12.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:12 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T03:39:12.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:12 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/2189767594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T03:39:14.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:14 smithi169 ceph-mon[35668]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:14.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:14 smithi112 ceph-mon[27737]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:16.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:16 smithi112 ceph-mon[27737]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:16.458 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:16 smithi169 ceph-mon[35668]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:18.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:18 smithi169 ceph-mon[35668]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:18.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:18 smithi112 ceph-mon[27737]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:20.513 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:20 smithi169 ceph-mon[35668]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:20.513 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:20 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:39:20.514 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:20 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-10T03:39:20.514 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:20 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:39:20.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:20 smithi112 ceph-mon[27737]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:20.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:20 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:39:20.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:20 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-10T03:39:20.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:20 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:39:21.318 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:21 smithi169 ceph-mon[35668]: Deploying daemon osd.5 on smithi169 2024-09-10T03:39:21.506 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:21 smithi112 ceph-mon[27737]: Deploying daemon osd.5 on smithi169 2024-09-10T03:39:22.500 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:22 smithi169 ceph-mon[35668]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:22.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:22 smithi112 ceph-mon[27737]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:23.472 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:23 smithi169 ceph-mon[35668]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:23.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:23 smithi112 ceph-mon[27737]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:26.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:25 smithi169 ceph-mon[35668]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:26.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:25 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:39:26.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:25 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:26.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:25 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:26.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:25 smithi112 ceph-mon[27737]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:26.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:25 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:39:26.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:25 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:26.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:25 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:28.258 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:27 smithi169 ceph-mon[35668]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:28.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:27 smithi112 ceph-mon[27737]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:29.843 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:29.842+0000 7f8a5effd640 1 -- 172.21.15.169:0/490058304 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8a6806aab0 con 0x7f8a5406de30 2024-09-10T03:39:29.843 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 5 on host 'smithi169' 2024-09-10T03:39:29.845 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:29.845+0000 7f8a6fde2640 1 -- 172.21.15.169:0/490058304 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f8a5406de30 msgr2=0x7f8a540702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:29.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:29.845+0000 7f8a6fde2640 1 --2- 172.21.15.169:0/490058304 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f8a5406de30 0x7f8a540702f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8a58000c00 tx=0x7f8a580058d0 comp rx=0 tx=0).stop 2024-09-10T03:39:29.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:29.845+0000 7f8a6fde2640 1 -- 172.21.15.169:0/490058304 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a68100880 msgr2=0x7f8a68199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:29.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:29.845+0000 7f8a6fde2640 1 --2- 172.21.15.169:0/490058304 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a68100880 0x7f8a68199760 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400b3d0 tx=0x7f8a6400b890 comp rx=0 tx=0).stop 2024-09-10T03:39:29.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:29.845+0000 7f8a6fde2640 1 -- 172.21.15.169:0/490058304 shutdown_connections 2024-09-10T03:39:29.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:29.845+0000 7f8a6fde2640 1 --2- 172.21.15.169:0/490058304 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f8a5406de30 0x7f8a540702f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:29.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:29.845+0000 7f8a6fde2640 1 --2- 172.21.15.169:0/490058304 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a68101a80 0x7f8a68199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:29.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:29.845+0000 7f8a6fde2640 1 --2- 172.21.15.169:0/490058304 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a68100880 0x7f8a68199760 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:29.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:29.845+0000 7f8a6fde2640 1 -- 172.21.15.169:0/490058304 >> 172.21.15.169:0/490058304 conn(0x7f8a680fc010 msgr2=0x7f8a6806a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:29.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:29.845+0000 7f8a6fde2640 1 -- 172.21.15.169:0/490058304 shutdown_connections 2024-09-10T03:39:29.846 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:29.845+0000 7f8a6fde2640 1 -- 172.21.15.169:0/490058304 wait complete. 2024-09-10T03:39:30.145 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:29 smithi169 ceph-mon[35668]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:30.145 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:29 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:30.145 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:29 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:30.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:29 smithi112 ceph-mon[27737]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:30.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:29 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:30.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:29 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:30.482 DEBUG:teuthology.orchestra.run.smithi169:osd.5> sudo journalctl -f -n 0 -u ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@osd.5.service 2024-09-10T03:39:30.484 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-09-10T03:39:30.485 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd stat -f json 2024-09-10T03:39:30.752 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.194+0000 7f98a011d640 1 -- 172.21.15.112:0/2574365041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9898104d80 msgr2=0x7f9898107170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.194+0000 7f98a011d640 1 --2- 172.21.15.112:0/2574365041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9898104d80 0x7f9898107170 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9888009880 tx=0x7f988802a4a0 comp rx=0 tx=0).stop 2024-09-10T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.194+0000 7f98a011d640 1 -- 172.21.15.112:0/2574365041 shutdown_connections 2024-09-10T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.194+0000 7f98a011d640 1 --2- 172.21.15.112:0/2574365041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9898104d80 0x7f9898107170 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.194+0000 7f98a011d640 1 --2- 172.21.15.112:0/2574365041 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9898102450 0x7f9898104840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.194+0000 7f98a011d640 1 -- 172.21.15.112:0/2574365041 >> 172.21.15.112:0/2574365041 conn(0x7f98980fbff0 msgr2=0x7f98980fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.194+0000 7f98a011d640 1 -- 172.21.15.112:0/2574365041 shutdown_connections 2024-09-10T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.194+0000 7f98a011d640 1 -- 172.21.15.112:0/2574365041 wait complete. 2024-09-10T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.194+0000 7f98a011d640 1 Processor -- start 2024-09-10T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.194+0000 7f98a011d640 1 -- start start 2024-09-10T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.195+0000 7f98a011d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9898102450 0x7f9898101d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.195+0000 7f98a011d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9898104d80 0x7f9898100400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.195+0000 7f98a011d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f989810a780 con 0x7f9898102450 2024-09-10T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.195+0000 7f98a011d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f989810a600 con 0x7f9898104d80 2024-09-10T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.195+0000 7f989deb9640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9898102450 0x7f9898101d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.195+0000 7f989deb9640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9898102450 0x7f9898101d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:35642/0 (socket says 172.21.15.112:35642) 2024-09-10T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.195+0000 7f989deb9640 1 -- 172.21.15.112:0/546889113 learned_addr learned my addr 172.21.15.112:0/546889113 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.195+0000 7f989d6b8640 1 --2- 172.21.15.112:0/546889113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9898104d80 0x7f9898100400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.195+0000 7f989deb9640 1 -- 172.21.15.112:0/546889113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9898104d80 msgr2=0x7f9898100400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.195+0000 7f989deb9640 1 --2- 172.21.15.112:0/546889113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9898104d80 0x7f9898100400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.195+0000 7f989deb9640 1 -- 172.21.15.112:0/546889113 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9898100940 con 0x7f9898102450 2024-09-10T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.195+0000 7f989deb9640 1 --2- 172.21.15.112:0/546889113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9898102450 0x7f9898101d40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f989400d6f0 tx=0x7f989400dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.195+0000 7f988effd640 1 -- 172.21.15.112:0/546889113 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f9894017070 con 0x7f9898102450 2024-09-10T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.195+0000 7f98a011d640 1 -- 172.21.15.112:0/546889113 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9898100c30 con 0x7f9898102450 2024-09-10T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.195+0000 7f988effd640 1 -- 172.21.15.112:0/546889113 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f989400de00 con 0x7f9898102450 2024-09-10T03:39:31.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.195+0000 7f98a011d640 1 -- 172.21.15.112:0/546889113 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9898101190 con 0x7f9898102450 2024-09-10T03:39:31.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.196+0000 7f988effd640 1 -- 172.21.15.112:0/546889113 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f9894004bb0 con 0x7f9898102450 2024-09-10T03:39:31.199 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.197+0000 7f98a011d640 1 -- 172.21.15.112:0/546889113 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f989810af60 con 0x7f9898102450 2024-09-10T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.197+0000 7f988effd640 1 -- 172.21.15.112:0/546889113 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f9894004e60 con 0x7f9898102450 2024-09-10T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.197+0000 7f988effd640 1 --2- 172.21.15.112:0/546889113 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f988406de30 0x7f98840702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.198+0000 7f988effd640 1 -- 172.21.15.112:0/546889113 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4321+0+0 (secure 0 0 0) 0x7f989408d2d0 con 0x7f9898102450 2024-09-10T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.198+0000 7f989d6b8640 1 --2- 172.21.15.112:0/546889113 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f988406de30 0x7f98840702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.198+0000 7f989d6b8640 1 --2- 172.21.15.112:0/546889113 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f988406de30 0x7f98840702f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9888009230 tx=0x7f98880058d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:31.203 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.202+0000 7f988effd640 1 -- 172.21.15.112:0/546889113 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f989405ef90 con 0x7f9898102450 2024-09-10T03:39:31.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.371+0000 7f98a011d640 1 -- 172.21.15.112:0/546889113 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f9898102830 con 0x7f9898102450 2024-09-10T03:39:31.372 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.371+0000 7f988effd640 1 -- 172.21.15.112:0/546889113 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v35) ==== 74+0+130 (secure 0 0 0) 0x7f98940637c0 con 0x7f9898102450 2024-09-10T03:39:31.372 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:39:31.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.373+0000 7f98a011d640 1 -- 172.21.15.112:0/546889113 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f988406de30 msgr2=0x7f98840702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:31.374 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.373+0000 7f98a011d640 1 --2- 172.21.15.112:0/546889113 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f988406de30 0x7f98840702f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9888009230 tx=0x7f98880058d0 comp rx=0 tx=0).stop 2024-09-10T03:39:31.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.373+0000 7f98a011d640 1 -- 172.21.15.112:0/546889113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9898102450 msgr2=0x7f9898101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:31.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.373+0000 7f98a011d640 1 --2- 172.21.15.112:0/546889113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9898102450 0x7f9898101d40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f989400d6f0 tx=0x7f989400dbb0 comp rx=0 tx=0).stop 2024-09-10T03:39:31.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.373+0000 7f98a011d640 1 -- 172.21.15.112:0/546889113 shutdown_connections 2024-09-10T03:39:31.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.373+0000 7f98a011d640 1 --2- 172.21.15.112:0/546889113 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f988406de30 0x7f98840702f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:31.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.373+0000 7f98a011d640 1 --2- 172.21.15.112:0/546889113 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9898104d80 0x7f9898100400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:31.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.373+0000 7f98a011d640 1 --2- 172.21.15.112:0/546889113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9898102450 0x7f9898101d40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:31.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.373+0000 7f98a011d640 1 -- 172.21.15.112:0/546889113 >> 172.21.15.112:0/546889113 conn(0x7f98980fbff0 msgr2=0x7f98980fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:31.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.373+0000 7f98a011d640 1 -- 172.21.15.112:0/546889113 shutdown_connections 2024-09-10T03:39:31.375 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:31.373+0000 7f98a011d640 1 -- 172.21.15.112:0/546889113 wait complete. 2024-09-10T03:39:31.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:31 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:31.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:31 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:31.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:31 smithi112 ceph-mon[27737]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:31.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:31 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/546889113' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T03:39:31.909 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:31 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:31.909 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:31 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:31.909 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:31 smithi169 ceph-mon[35668]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:31.909 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:31 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/546889113' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T03:39:32.145 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":35,"num_osds":6,"num_up_osds":5,"osd_up_since":1725939548,"num_in_osds":6,"osd_in_since":1725939551,"num_remapped_pgs":0} 2024-09-10T03:39:32.247 INFO:journalctl@ceph.osd.5.smithi169.stdout:Sep 10 03:39:31 smithi169 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-5[55154]: 2024-09-10T03:39:31.908+0000 7f5664ff9640 -1 osd.5 0 log_to_monitors true 2024-09-10T03:39:32.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:32 smithi112 ceph-mon[27737]: from='osd.5 [v2:172.21.15.169:6816/893753355,v1:172.21.15.169:6817/893753355]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T03:39:32.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:32 smithi112 ceph-mon[27737]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T03:39:32.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:32 smithi169 ceph-mon[35668]: from='osd.5 [v2:172.21.15.169:6816/893753355,v1:172.21.15.169:6817/893753355]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T03:39:32.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:32 smithi169 ceph-mon[35668]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T03:39:33.146 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd stat -f json 2024-09-10T03:39:33.394 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:33.770 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:33 smithi112 ceph-mon[27737]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-10T03:39:33.770 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:33 smithi112 ceph-mon[27737]: from='osd.5 [v2:172.21.15.169:6816/893753355,v1:172.21.15.169:6817/893753355]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-09-10T03:39:33.770 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:33 smithi112 ceph-mon[27737]: osdmap e36: 6 total, 5 up, 6 in 2024-09-10T03:39:33.770 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:33 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T03:39:33.771 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:33 smithi112 ceph-mon[27737]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-09-10T03:39:33.771 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:33 smithi112 ceph-mon[27737]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:33.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.849+0000 7f13324df640 1 -- 172.21.15.112:0/3101549381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f132c100840 msgr2=0x7f132c100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:33.878 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.849+0000 7f13324df640 1 --2- 172.21.15.112:0/3101549381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f132c100840 0x7f132c100c40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f131c009880 tx=0x7f131c02a4a0 comp rx=0 tx=0).stop 2024-09-10T03:39:33.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.849+0000 7f13324df640 1 -- 172.21.15.112:0/3101549381 shutdown_connections 2024-09-10T03:39:33.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.849+0000 7f13324df640 1 --2- 172.21.15.112:0/3101549381 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f132c101a40 0x7f132c101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:33.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.849+0000 7f13324df640 1 --2- 172.21.15.112:0/3101549381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f132c100840 0x7f132c100c40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:33.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.849+0000 7f13324df640 1 -- 172.21.15.112:0/3101549381 >> 172.21.15.112:0/3101549381 conn(0x7f132c0fbff0 msgr2=0x7f132c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:33.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.849+0000 7f13324df640 1 -- 172.21.15.112:0/3101549381 shutdown_connections 2024-09-10T03:39:33.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.849+0000 7f13324df640 1 -- 172.21.15.112:0/3101549381 wait complete. 2024-09-10T03:39:33.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.850+0000 7f13324df640 1 Processor -- start 2024-09-10T03:39:33.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.850+0000 7f13324df640 1 -- start start 2024-09-10T03:39:33.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.850+0000 7f13324df640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f132c100840 0x7f132c06a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:33.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.850+0000 7f13324df640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f132c101a40 0x7f132c19c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:33.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.850+0000 7f13324df640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f132c109660 con 0x7f132c100840 2024-09-10T03:39:33.879 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.850+0000 7f13324df640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f132c1094e0 con 0x7f132c101a40 2024-09-10T03:39:33.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.851+0000 7f132bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f132c100840 0x7f132c06a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:33.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.851+0000 7f132bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f132c100840 0x7f132c06a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:35668/0 (socket says 172.21.15.112:35668) 2024-09-10T03:39:33.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.851+0000 7f132bfff640 1 -- 172.21.15.112:0/96794846 learned_addr learned my addr 172.21.15.112:0/96794846 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:33.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.851+0000 7f132b7fe640 1 --2- 172.21.15.112:0/96794846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f132c101a40 0x7f132c19c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:33.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.851+0000 7f132bfff640 1 -- 172.21.15.112:0/96794846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f132c101a40 msgr2=0x7f132c19c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:33.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.851+0000 7f132bfff640 1 --2- 172.21.15.112:0/96794846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f132c101a40 0x7f132c19c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:33.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.851+0000 7f132bfff640 1 -- 172.21.15.112:0/96794846 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c19c7b0 con 0x7f132c100840 2024-09-10T03:39:33.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.851+0000 7f132bfff640 1 --2- 172.21.15.112:0/96794846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f132c100840 0x7f132c06a620 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f131c006010 tx=0x7f131c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:33.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.851+0000 7f13297fa640 1 -- 172.21.15.112:0/96794846 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f131c03f070 con 0x7f132c100840 2024-09-10T03:39:33.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.851+0000 7f13297fa640 1 -- 172.21.15.112:0/96794846 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f131c004a50 con 0x7f132c100840 2024-09-10T03:39:33.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.851+0000 7f13324df640 1 -- 172.21.15.112:0/96794846 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f132c19ca40 con 0x7f132c100840 2024-09-10T03:39:33.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.851+0000 7f13297fa640 1 -- 172.21.15.112:0/96794846 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f131c035030 con 0x7f132c100840 2024-09-10T03:39:33.880 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.851+0000 7f13324df640 1 -- 172.21.15.112:0/96794846 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f132c19cf70 con 0x7f132c100840 2024-09-10T03:39:33.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.852+0000 7f13324df640 1 -- 172.21.15.112:0/96794846 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f132c100c40 con 0x7f132c100840 2024-09-10T03:39:33.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.853+0000 7f13297fa640 1 -- 172.21.15.112:0/96794846 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f131c042070 con 0x7f132c100840 2024-09-10T03:39:33.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.853+0000 7f13297fa640 1 --2- 172.21.15.112:0/96794846 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f131406de30 0x7f13140702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:33.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.854+0000 7f13297fa640 1 -- 172.21.15.112:0/96794846 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 4358+0+0 (secure 0 0 0) 0x7f131c0ae2a0 con 0x7f132c100840 2024-09-10T03:39:33.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.854+0000 7f132b7fe640 1 --2- 172.21.15.112:0/96794846 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f131406de30 0x7f13140702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:33.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.854+0000 7f132b7fe640 1 --2- 172.21.15.112:0/96794846 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f131406de30 0x7f13140702f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f132c06a3a0 tx=0x7f132000b410 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:33.881 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:33.858+0000 7f13297fa640 1 -- 172.21.15.112:0/96794846 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f131c004c00 con 0x7f132c100840 2024-09-10T03:39:33.987 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:33 smithi169 ceph-mon[35668]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-10T03:39:33.987 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:33 smithi169 ceph-mon[35668]: from='osd.5 [v2:172.21.15.169:6816/893753355,v1:172.21.15.169:6817/893753355]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-09-10T03:39:33.988 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:33 smithi169 ceph-mon[35668]: osdmap e36: 6 total, 5 up, 6 in 2024-09-10T03:39:33.988 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:33 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T03:39:33.988 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:33 smithi169 ceph-mon[35668]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2024-09-10T03:39:33.988 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:33 smithi169 ceph-mon[35668]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:34.035 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:34.033+0000 7f13324df640 1 -- 172.21.15.112:0/96794846 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f132c062df0 con 0x7f132c100840 2024-09-10T03:39:34.036 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:34.034+0000 7f13297fa640 1 -- 172.21.15.112:0/96794846 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v37) ==== 74+0+130 (secure 0 0 0) 0x7f131c0846e0 con 0x7f132c100840 2024-09-10T03:39:34.036 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:39:34.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:34.036+0000 7f13324df640 1 -- 172.21.15.112:0/96794846 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f131406de30 msgr2=0x7f13140702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:34.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:34.036+0000 7f13324df640 1 --2- 172.21.15.112:0/96794846 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f131406de30 0x7f13140702f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f132c06a3a0 tx=0x7f132000b410 comp rx=0 tx=0).stop 2024-09-10T03:39:34.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:34.036+0000 7f13324df640 1 -- 172.21.15.112:0/96794846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f132c100840 msgr2=0x7f132c06a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:34.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:34.036+0000 7f13324df640 1 --2- 172.21.15.112:0/96794846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f132c100840 0x7f132c06a620 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f131c006010 tx=0x7f131c0045e0 comp rx=0 tx=0).stop 2024-09-10T03:39:34.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:34.036+0000 7f13324df640 1 -- 172.21.15.112:0/96794846 shutdown_connections 2024-09-10T03:39:34.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:34.036+0000 7f13324df640 1 --2- 172.21.15.112:0/96794846 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f131406de30 0x7f13140702f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:34.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:34.036+0000 7f13324df640 1 --2- 172.21.15.112:0/96794846 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f132c101a40 0x7f132c19c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:34.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:34.036+0000 7f13324df640 1 --2- 172.21.15.112:0/96794846 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f132c100840 0x7f132c06a620 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:34.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:34.036+0000 7f13324df640 1 -- 172.21.15.112:0/96794846 >> 172.21.15.112:0/96794846 conn(0x7f132c0fbff0 msgr2=0x7f132c104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:34.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:34.036+0000 7f13324df640 1 -- 172.21.15.112:0/96794846 shutdown_connections 2024-09-10T03:39:34.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:34.036+0000 7f13324df640 1 -- 172.21.15.112:0/96794846 wait complete. 2024-09-10T03:39:34.593 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-09-10T03:39:34.594 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: osdmap e37: 6 total, 5 up, 6 in 2024-09-10T03:39:34.594 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T03:39:34.594 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T03:39:34.594 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: Detected new or changed devices on smithi169 2024-09-10T03:39:34.594 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:34.594 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:34.594 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:39:34.594 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:39:34.594 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:39:34.594 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: Adjusting osd_memory_target on smithi169 to 4970M 2024-09-10T03:39:34.594 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:34.594 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:39:34.594 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:39:34.595 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:34.595 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:39:34.595 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:39:34.595 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:39:34.595 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:34.595 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/96794846' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T03:39:34.595 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:34 smithi112 ceph-mon[27737]: from='osd.5 ' entity='osd.5' 2024-09-10T03:39:34.621 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":37,"num_osds":6,"num_up_osds":5,"osd_up_since":1725939548,"num_in_osds":6,"osd_in_since":1725939551,"num_remapped_pgs":0} 2024-09-10T03:39:34.776 INFO:journalctl@ceph.osd.5.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-5[55154]: 2024-09-10T03:39:34.364+0000 7f5661790640 -1 osd.5 0 waiting for initial osdmap 2024-09-10T03:39:34.776 INFO:journalctl@ceph.osd.5.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-5[55154]: 2024-09-10T03:39:34.373+0000 7f5658d8b640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T03:39:34.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2024-09-10T03:39:34.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: osdmap e37: 6 total, 5 up, 6 in 2024-09-10T03:39:34.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T03:39:34.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T03:39:34.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: Detected new or changed devices on smithi169 2024-09-10T03:39:34.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:34.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:34.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:39:34.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:39:34.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T03:39:34.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: Adjusting osd_memory_target on smithi169 to 4970M 2024-09-10T03:39:34.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:34.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:39:34.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:39:34.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:34.778 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:39:34.778 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:39:34.778 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:39:34.778 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:39:34.778 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/96794846' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T03:39:34.778 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:34 smithi169 ceph-mon[35668]: from='osd.5 ' entity='osd.5' 2024-09-10T03:39:35.623 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd stat -f json 2024-09-10T03:39:35.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:35 smithi112 ceph-mon[27737]: purged_snaps scrub starts 2024-09-10T03:39:35.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:35 smithi112 ceph-mon[27737]: purged_snaps scrub ok 2024-09-10T03:39:35.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T03:39:35.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:35 smithi112 ceph-mon[27737]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:35.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:39:35.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:35 smithi112 ceph-mon[27737]: osd.5 [v2:172.21.15.169:6816/893753355,v1:172.21.15.169:6817/893753355] boot 2024-09-10T03:39:35.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:35 smithi112 ceph-mon[27737]: osdmap e38: 6 total, 6 up, 6 in 2024-09-10T03:39:35.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T03:39:35.881 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:36.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:35 smithi169 ceph-mon[35668]: purged_snaps scrub starts 2024-09-10T03:39:36.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:35 smithi169 ceph-mon[35668]: purged_snaps scrub ok 2024-09-10T03:39:36.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T03:39:36.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:35 smithi169 ceph-mon[35668]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-09-10T03:39:36.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:39:36.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:35 smithi169 ceph-mon[35668]: osd.5 [v2:172.21.15.169:6816/893753355,v1:172.21.15.169:6817/893753355] boot 2024-09-10T03:39:36.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:35 smithi169 ceph-mon[35668]: osdmap e38: 6 total, 6 up, 6 in 2024-09-10T03:39:36.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T03:39:36.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.316+0000 7fa01becc640 1 -- 172.21.15.112:0/3453449343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa014102b90 msgr2=0x7fa014103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:36.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.316+0000 7fa01becc640 1 --2- 172.21.15.112:0/3453449343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa014102b90 0x7fa014103010 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa008009880 tx=0x7fa00802a4a0 comp rx=0 tx=0).stop 2024-09-10T03:39:36.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.316+0000 7fa01becc640 1 -- 172.21.15.112:0/3453449343 shutdown_connections 2024-09-10T03:39:36.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.316+0000 7fa01becc640 1 --2- 172.21.15.112:0/3453449343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa014102b90 0x7fa014103010 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:36.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.316+0000 7fa01becc640 1 --2- 172.21.15.112:0/3453449343 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa014101990 0x7fa014101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:36.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.316+0000 7fa01becc640 1 -- 172.21.15.112:0/3453449343 >> 172.21.15.112:0/3453449343 conn(0x7fa0140fd140 msgr2=0x7fa0140ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:36.333 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.316+0000 7fa01becc640 1 -- 172.21.15.112:0/3453449343 shutdown_connections 2024-09-10T03:39:36.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.316+0000 7fa01becc640 1 -- 172.21.15.112:0/3453449343 wait complete. 2024-09-10T03:39:36.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.317+0000 7fa01becc640 1 Processor -- start 2024-09-10T03:39:36.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.317+0000 7fa01becc640 1 -- start start 2024-09-10T03:39:36.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.317+0000 7fa01becc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa014101990 0x7fa014075f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:36.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.317+0000 7fa01becc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa014102b90 0x7fa0140745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:36.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.317+0000 7fa01becc640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa01410a7b0 con 0x7fa014102b90 2024-09-10T03:39:36.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.317+0000 7fa01becc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fa01410a630 con 0x7fa014101990 2024-09-10T03:39:36.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.317+0000 7fa019467640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa014102b90 0x7fa0140745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:36.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.317+0000 7fa019467640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa014102b90 0x7fa0140745e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:44836/0 (socket says 172.21.15.112:44836) 2024-09-10T03:39:36.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.317+0000 7fa019467640 1 -- 172.21.15.112:0/4274631000 learned_addr learned my addr 172.21.15.112:0/4274631000 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:36.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.317+0000 7fa019c68640 1 --2- 172.21.15.112:0/4274631000 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa014101990 0x7fa014075f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:36.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.317+0000 7fa019467640 1 -- 172.21.15.112:0/4274631000 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa014101990 msgr2=0x7fa014075f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:36.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.317+0000 7fa019467640 1 --2- 172.21.15.112:0/4274631000 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa014101990 0x7fa014075f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:36.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.317+0000 7fa019467640 1 -- 172.21.15.112:0/4274631000 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa014074b20 con 0x7fa014102b90 2024-09-10T03:39:36.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.318+0000 7fa019467640 1 --2- 172.21.15.112:0/4274631000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa014102b90 0x7fa0140745e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa008006010 tx=0x7fa0080048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:36.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.318+0000 7fa012ffd640 1 -- 172.21.15.112:0/4274631000 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fa00803f070 con 0x7fa014102b90 2024-09-10T03:39:36.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.318+0000 7fa01becc640 1 -- 172.21.15.112:0/4274631000 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa014074db0 con 0x7fa014102b90 2024-09-10T03:39:36.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.318+0000 7fa012ffd640 1 -- 172.21.15.112:0/4274631000 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa0080044d0 con 0x7fa014102b90 2024-09-10T03:39:36.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.318+0000 7fa012ffd640 1 -- 172.21.15.112:0/4274631000 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fa008035030 con 0x7fa014102b90 2024-09-10T03:39:36.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.318+0000 7fa01becc640 1 -- 172.21.15.112:0/4274631000 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0140752e0 con 0x7fa014102b90 2024-09-10T03:39:36.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.319+0000 7fa01becc640 1 -- 172.21.15.112:0/4274631000 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa01410af90 con 0x7fa014102b90 2024-09-10T03:39:36.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.320+0000 7fa012ffd640 1 -- 172.21.15.112:0/4274631000 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fa008042070 con 0x7fa014102b90 2024-09-10T03:39:36.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.320+0000 7fa012ffd640 1 --2- 172.21.15.112:0/4274631000 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f9ffc06de30 0x7f9ffc0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:36.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.320+0000 7fa012ffd640 1 -- 172.21.15.112:0/4274631000 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 4638+0+0 (secure 0 0 0) 0x7fa0080ae540 con 0x7fa014102b90 2024-09-10T03:39:36.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.320+0000 7fa019c68640 1 --2- 172.21.15.112:0/4274631000 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f9ffc06de30 0x7f9ffc0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:36.335 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.321+0000 7fa019c68640 1 --2- 172.21.15.112:0/4274631000 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f9ffc06de30 0x7f9ffc0702f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa0141029f0 tx=0x7fa00c005f90 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:36.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.324+0000 7fa012ffd640 1 -- 172.21.15.112:0/4274631000 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa0080800d0 con 0x7fa014102b90 2024-09-10T03:39:36.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.489+0000 7fa01becc640 1 -- 172.21.15.112:0/4274631000 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fa01419f500 con 0x7fa014102b90 2024-09-10T03:39:36.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.489+0000 7fa012ffd640 1 -- 172.21.15.112:0/4274631000 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) ==== 74+0+130 (secure 0 0 0) 0x7fa0080ae360 con 0x7fa014102b90 2024-09-10T03:39:36.492 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:39:36.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.491+0000 7fa01becc640 1 -- 172.21.15.112:0/4274631000 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f9ffc06de30 msgr2=0x7f9ffc0702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:36.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.491+0000 7fa01becc640 1 --2- 172.21.15.112:0/4274631000 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f9ffc06de30 0x7f9ffc0702f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa0141029f0 tx=0x7fa00c005f90 comp rx=0 tx=0).stop 2024-09-10T03:39:36.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.491+0000 7fa01becc640 1 -- 172.21.15.112:0/4274631000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa014102b90 msgr2=0x7fa0140745e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:36.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.491+0000 7fa01becc640 1 --2- 172.21.15.112:0/4274631000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa014102b90 0x7fa0140745e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa008006010 tx=0x7fa0080048f0 comp rx=0 tx=0).stop 2024-09-10T03:39:36.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.491+0000 7fa01becc640 1 -- 172.21.15.112:0/4274631000 shutdown_connections 2024-09-10T03:39:36.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.491+0000 7fa01becc640 1 --2- 172.21.15.112:0/4274631000 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f9ffc06de30 0x7f9ffc0702f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:36.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.491+0000 7fa01becc640 1 --2- 172.21.15.112:0/4274631000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa014102b90 0x7fa0140745e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:36.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.491+0000 7fa01becc640 1 --2- 172.21.15.112:0/4274631000 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa014101990 0x7fa014075f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:36.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.491+0000 7fa01becc640 1 -- 172.21.15.112:0/4274631000 >> 172.21.15.112:0/4274631000 conn(0x7fa0140fd140 msgr2=0x7fa0140ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:36.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.492+0000 7fa01becc640 1 -- 172.21.15.112:0/4274631000 shutdown_connections 2024-09-10T03:39:36.494 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:36.492+0000 7fa01becc640 1 -- 172.21.15.112:0/4274631000 wait complete. 2024-09-10T03:39:36.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:36 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/4274631000' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T03:39:37.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:36 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/4274631000' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T03:39:37.247 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":38,"num_osds":6,"num_up_osds":6,"osd_up_since":1725939575,"num_in_osds":6,"osd_in_since":1725939551,"num_remapped_pgs":0} 2024-09-10T03:39:37.248 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T03:39:37.507 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:37.879 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:37 smithi112 ceph-mon[27737]: osdmap e39: 6 total, 6 up, 6 in 2024-09-10T03:39:37.879 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:37 smithi112 ceph-mon[27737]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:37.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.956+0000 7fbe9b688640 1 -- 172.21.15.112:0/3188556577 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbe94101930 msgr2=0x7fbe94101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:37.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.956+0000 7fbe9b688640 1 --2- 172.21.15.112:0/3188556577 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbe94101930 0x7fbe94101d90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbe88009880 tx=0x7fbe8802a4a0 comp rx=0 tx=0).stop 2024-09-10T03:39:37.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.956+0000 7fbe9b688640 1 -- 172.21.15.112:0/3188556577 shutdown_connections 2024-09-10T03:39:37.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.956+0000 7fbe9b688640 1 --2- 172.21.15.112:0/3188556577 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbe94101930 0x7fbe94101d90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:37.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.956+0000 7fbe9b688640 1 --2- 172.21.15.112:0/3188556577 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe94107930 0x7fbe94107d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:37.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.956+0000 7fbe9b688640 1 -- 172.21.15.112:0/3188556577 >> 172.21.15.112:0/3188556577 conn(0x7fbe940fd630 msgr2=0x7fbe940ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:37.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.956+0000 7fbe9b688640 1 -- 172.21.15.112:0/3188556577 shutdown_connections 2024-09-10T03:39:37.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.956+0000 7fbe9b688640 1 -- 172.21.15.112:0/3188556577 wait complete. 2024-09-10T03:39:37.987 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.957+0000 7fbe9b688640 1 Processor -- start 2024-09-10T03:39:37.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.957+0000 7fbe9b688640 1 -- start start 2024-09-10T03:39:37.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.957+0000 7fbe9b688640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbe94101930 0x7fbe9419d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:37.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.957+0000 7fbe9b688640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe94107930 0x7fbe9419dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:37.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.957+0000 7fbe9b688640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9410ee60 con 0x7fbe94101930 2024-09-10T03:39:37.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.957+0000 7fbe9b688640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9410ece0 con 0x7fbe94107930 2024-09-10T03:39:37.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.957+0000 7fbe99424640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbe94101930 0x7fbe9419d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:37.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.957+0000 7fbe99424640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbe94101930 0x7fbe9419d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:44854/0 (socket says 172.21.15.112:44854) 2024-09-10T03:39:37.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.957+0000 7fbe99424640 1 -- 172.21.15.112:0/836485110 learned_addr learned my addr 172.21.15.112:0/836485110 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:37.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.957+0000 7fbe99424640 1 -- 172.21.15.112:0/836485110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe94107930 msgr2=0x7fbe9419dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:37.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.957+0000 7fbe98c23640 1 --2- 172.21.15.112:0/836485110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe94107930 0x7fbe9419dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:37.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.957+0000 7fbe99424640 1 --2- 172.21.15.112:0/836485110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe94107930 0x7fbe9419dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:37.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.957+0000 7fbe99424640 1 -- 172.21.15.112:0/836485110 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe94197870 con 0x7fbe94101930 2024-09-10T03:39:37.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.957+0000 7fbe98c23640 1 --2- 172.21.15.112:0/836485110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe94107930 0x7fbe9419dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T03:39:37.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.958+0000 7fbe99424640 1 --2- 172.21.15.112:0/836485110 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbe94101930 0x7fbe9419d6b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00b420 tx=0x7fbe8c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:37.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.958+0000 7fbe927fc640 1 -- 172.21.15.112:0/836485110 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fbe8c011630 con 0x7fbe94101930 2024-09-10T03:39:37.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.958+0000 7fbe9b688640 1 -- 172.21.15.112:0/836485110 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe94197b60 con 0x7fbe94101930 2024-09-10T03:39:37.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.958+0000 7fbe927fc640 1 -- 172.21.15.112:0/836485110 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbe8c016070 con 0x7fbe94101930 2024-09-10T03:39:37.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.958+0000 7fbe927fc640 1 -- 172.21.15.112:0/836485110 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fbe8c010d40 con 0x7fbe94101930 2024-09-10T03:39:37.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.958+0000 7fbe9b688640 1 -- 172.21.15.112:0/836485110 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe941980f0 con 0x7fbe94101930 2024-09-10T03:39:37.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.959+0000 7fbe9b688640 1 -- 172.21.15.112:0/836485110 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe94103070 con 0x7fbe94101930 2024-09-10T03:39:37.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.959+0000 7fbe927fc640 1 -- 172.21.15.112:0/836485110 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fbe8c01b460 con 0x7fbe94101930 2024-09-10T03:39:37.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.960+0000 7fbe927fc640 1 --2- 172.21.15.112:0/836485110 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fbe7c06de30 0x7fbe7c0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:37.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.960+0000 7fbe98c23640 1 --2- 172.21.15.112:0/836485110 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fbe7c06de30 0x7fbe7c0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:37.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.960+0000 7fbe927fc640 1 -- 172.21.15.112:0/836485110 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7fbe8c08d0b0 con 0x7fbe94101930 2024-09-10T03:39:37.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.960+0000 7fbe98c23640 1 --2- 172.21.15.112:0/836485110 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fbe7c06de30 0x7fbe7c0702f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbe88031460 tx=0x7fbe880095f0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:37.989 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:37.964+0000 7fbe927fc640 1 -- 172.21.15.112:0/836485110 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbe8c05af90 con 0x7fbe94101930 2024-09-10T03:39:38.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:37 smithi169 ceph-mon[35668]: osdmap e39: 6 total, 6 up, 6 in 2024-09-10T03:39:38.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:37 smithi169 ceph-mon[35668]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:38.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:38.131+0000 7fbe9b688640 1 -- 172.21.15.112:0/836485110 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe94198da0 con 0x7fbe94101930 2024-09-10T03:39:38.134 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:38.132+0000 7fbe927fc640 1 -- 172.21.15.112:0/836485110 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+11420 (secure 0 0 0) 0x7fbe8c05ec40 con 0x7fbe94101930 2024-09-10T03:39:38.134 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:39:38.134 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":39,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","created":"2024-09-10T03:33:31.106241+0000","modified":"2024-09-10T03:39:36.567053+0000","last_up_change":"2024-09-10T03:39:35.366928+0000","last_in_change":"2024-09-10T03:39:11.309435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T03:38:15.123477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ae9a7fcc-9d61-4914-b88f-da4fc76d5b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":378527782},{"type":"v1","addr":"172.21.15.112:6803","nonce":378527782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":378527782},{"type":"v1","addr":"172.21.15.112:6805","nonce":378527782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":378527782},{"type":"v1","addr":"172.21.15.112:6809","nonce":378527782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":378527782},{"type":"v1","addr":"172.21.15.112:6807","nonce":378527782}]},"public_addr":"172.21.15.112:6803/378527782","cluster_addr":"172.21.15.112:6805/378527782","heartbeat_back_addr":"172.21.15.112:6809/378527782","heartbeat_front_addr":"172.21.15.112:6807/378527782","state":["exists","up"]},{"osd":1,"uuid":"41b3eaeb-7456-42dd-9405-0770a7129bc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":305082264},{"type":"v1","addr":"172.21.15.112:6811","nonce":305082264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":305082264},{"type":"v1","addr":"172.21.15.112:6813","nonce":305082264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":305082264},{"type":"v1","addr":"172.21.15.112:6817","nonce":305082264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":305082264},{"type":"v1","addr":"172.21.15.112:6815","nonce":305082264}]},"public_addr":"172.21.15.112:6811/305082264","cluster_addr":"172.21.15.112:6813/305082264","heartbeat_back_addr":"172.21.15.112:6817/305082264","heartbeat_front_addr":"172.21.15.112:6815/305082264","state":["exists","up"]},{"osd":2,"uuid":"e7d8e980-cf86-473c-805c-61ebb62e4d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3325040968},{"type":"v1","addr":"172.21.15.112:6819","nonce":3325040968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3325040968},{"type":"v1","addr":"172.21.15.112:6821","nonce":3325040968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3325040968},{"type":"v1","addr":"172.21.15.112:6825","nonce":3325040968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3325040968},{"type":"v1","addr":"172.21.15.112:6823","nonce":3325040968}]},"public_addr":"172.21.15.112:6819/3325040968","cluster_addr":"172.21.15.112:6821/3325040968","heartbeat_back_addr":"172.21.15.112:6825/3325040968","heartbeat_front_addr":"172.21.15.112:6823/3325040968","state":["exists","up"]},{"osd":3,"uuid":"eec71a12-c779-49af-a994-414f77d0ec78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3830555794},{"type":"v1","addr":"172.21.15.169:6801","nonce":3830555794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3830555794},{"type":"v1","addr":"172.21.15.169:6803","nonce":3830555794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":3830555794},{"type":"v1","addr":"172.21.15.169:6807","nonce":3830555794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":3830555794},{"type":"v1","addr":"172.21.15.169:6805","nonce":3830555794}]},"public_addr":"172.21.15.169:6801/3830555794","cluster_addr":"172.21.15.169:6803/3830555794","heartbeat_back_addr":"172.21.15.169:6807/3830555794","heartbeat_front_addr":"172.21.15.169:6805/3830555794","state":["exists","up"]},{"osd":4,"uuid":"77c831ef-a06d-4f4f-ad2a-c2cb04613fe0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":2285032153},{"type":"v1","addr":"172.21.15.169:6809","nonce":2285032153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":2285032153},{"type":"v1","addr":"172.21.15.169:6811","nonce":2285032153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":2285032153},{"type":"v1","addr":"172.21.15.169:6815","nonce":2285032153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":2285032153},{"type":"v1","addr":"172.21.15.169:6813","nonce":2285032153}]},"public_addr":"172.21.15.169:6809/2285032153","cluster_addr":"172.21.15.169:6811/2285032153","heartbeat_back_addr":"172.21.15.169:6815/2285032153","heartbeat_front_addr":"172.21.15.169:6813/2285032153","state":["exists","up"]},{"osd":5,"uuid":"346b5969-a50d-4340-a839-ef1085e248fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":893753355},{"type":"v1","addr":"172.21.15.169:6817","nonce":893753355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":893753355},{"type":"v1","addr":"172.21.15.169:6819","nonce":893753355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":893753355},{"type":"v1","addr":"172.21.15.169:6823","nonce":893753355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":893753355},{"type":"v1","addr":"172.21.15.169:6821","nonce":893753355}]},"public_addr":"172.21.15.169:6817/893753355","cluster_addr":"172.21.15.169:6819/893753355","heartbeat_back_addr":"172.21.15.169:6823/893753355","heartbeat_front_addr":"172.21.15.169:6821/893753355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:37:09.717391+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:37:43.057817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:38:12.048486+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:38:37.732503+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:39:06.199278+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:39:32.899049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.112:6801/171256542":"2024-09-11T03:35:34.814063+0000","172.21.15.112:0/3104768277":"2024-09-11T03:34:03.009905+0000","172.21.15.112:6800/171256542":"2024-09-11T03:35:34.814063+0000","172.21.15.112:6800/3423408536":"2024-09-11T03:34:03.009905+0000","172.21.15.112:0/3329867731":"2024-09-11T03:34:03.009905+0000","172.21.15.112:0/3444044994":"2024-09-11T03:34:03.009905+0000","172.21.15.112:0/288016527":"2024-09-11T03:34:33.723096+0000","172.21.15.112:0/3338248392":"2024-09-11T03:34:33.723096+0000","172.21.15.112:6801/4183006970":"2024-09-11T03:34:33.723096+0000","172.21.15.112:0/4078185439":"2024-09-11T03:34:33.723096+0000","172.21.15.112:6800/4183006970":"2024-09-11T03:34:33.723096+0000","172.21.15.112:0/3410252058":"2024-09-11T03:35:34.814063+0000","172.21.15.112:6801/3423408536":"2024-09-11T03:34:03.009905+0000","172.21.15.112:0/1394011979":"2024-09-11T03:35:34.814063+0000","172.21.15.112:0/522295971":"2024-09-11T03:35:34.814063+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T03:39:38.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:38.134+0000 7fbe9b688640 1 -- 172.21.15.112:0/836485110 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fbe7c06de30 msgr2=0x7fbe7c0702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:38.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:38.134+0000 7fbe9b688640 1 --2- 172.21.15.112:0/836485110 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fbe7c06de30 0x7fbe7c0702f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbe88031460 tx=0x7fbe880095f0 comp rx=0 tx=0).stop 2024-09-10T03:39:38.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:38.134+0000 7fbe9b688640 1 -- 172.21.15.112:0/836485110 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbe94101930 msgr2=0x7fbe9419d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:38.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:38.134+0000 7fbe9b688640 1 --2- 172.21.15.112:0/836485110 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbe94101930 0x7fbe9419d6b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00b420 tx=0x7fbe8c00b8e0 comp rx=0 tx=0).stop 2024-09-10T03:39:38.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:38.134+0000 7fbe9b688640 1 -- 172.21.15.112:0/836485110 shutdown_connections 2024-09-10T03:39:38.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:38.134+0000 7fbe9b688640 1 --2- 172.21.15.112:0/836485110 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fbe7c06de30 0x7fbe7c0702f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:38.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:38.134+0000 7fbe9b688640 1 --2- 172.21.15.112:0/836485110 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe94107930 0x7fbe9419dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:38.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:38.134+0000 7fbe9b688640 1 --2- 172.21.15.112:0/836485110 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbe94101930 0x7fbe9419d6b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:38.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:38.134+0000 7fbe9b688640 1 -- 172.21.15.112:0/836485110 >> 172.21.15.112:0/836485110 conn(0x7fbe940fd630 msgr2=0x7fbe940fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:38.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:38.134+0000 7fbe9b688640 1 -- 172.21.15.112:0/836485110 shutdown_connections 2024-09-10T03:39:38.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:38.134+0000 7fbe9b688640 1 -- 172.21.15.112:0/836485110 wait complete. 2024-09-10T03:39:38.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:38 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/836485110' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T03:39:38.912 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T03:38:15.123477+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-10T03:39:38.912 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-10T03:39:38.987 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:38 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/836485110' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T03:39:39.160 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:39.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.576+0000 7f6d05f38640 1 -- 172.21.15.112:0/2347758641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d001007e0 msgr2=0x7f6d00100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:39.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.576+0000 7f6d05f38640 1 --2- 172.21.15.112:0/2347758641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d001007e0 0x7f6d00100c40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009880 tx=0x7f6ce802a4a0 comp rx=0 tx=0).stop 2024-09-10T03:39:39.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.576+0000 7f6d05f38640 1 -- 172.21.15.112:0/2347758641 shutdown_connections 2024-09-10T03:39:39.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.576+0000 7f6d05f38640 1 --2- 172.21.15.112:0/2347758641 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d001007e0 0x7f6d00100c40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:39.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.576+0000 7f6d05f38640 1 --2- 172.21.15.112:0/2347758641 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d001067e0 0x7f6d00106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.576+0000 7f6d05f38640 1 -- 172.21.15.112:0/2347758641 >> 172.21.15.112:0/2347758641 conn(0x7f6d000fc4c0 msgr2=0x7f6d000fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.576+0000 7f6d05f38640 1 -- 172.21.15.112:0/2347758641 shutdown_connections 2024-09-10T03:39:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.576+0000 7f6d05f38640 1 -- 172.21.15.112:0/2347758641 wait complete. 2024-09-10T03:39:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.576+0000 7f6d05f38640 1 Processor -- start 2024-09-10T03:39:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.576+0000 7f6d05f38640 1 -- start start 2024-09-10T03:39:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.577+0000 7f6d05f38640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d001007e0 0x7f6d0019d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.577+0000 7f6d05f38640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d001067e0 0x7f6d0019dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.577+0000 7f6d05f38640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0010dd10 con 0x7f6d001007e0 2024-09-10T03:39:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.577+0000 7f6d05f38640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0010db90 con 0x7f6d001067e0 2024-09-10T03:39:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.577+0000 7f6cff7fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d001007e0 0x7f6d0019d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.577+0000 7f6cff7fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d001007e0 0x7f6d0019d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:44874/0 (socket says 172.21.15.112:44874) 2024-09-10T03:39:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.577+0000 7f6cff7fe640 1 -- 172.21.15.112:0/3941376412 learned_addr learned my addr 172.21.15.112:0/3941376412 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:39.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.577+0000 7f6cf7fff640 1 --2- 172.21.15.112:0/3941376412 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d001067e0 0x7f6d0019dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:39.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.577+0000 7f6cff7fe640 1 -- 172.21.15.112:0/3941376412 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d001067e0 msgr2=0x7f6d0019dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:39.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.577+0000 7f6cff7fe640 1 --2- 172.21.15.112:0/3941376412 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d001067e0 0x7f6d0019dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:39.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.577+0000 7f6cff7fe640 1 -- 172.21.15.112:0/3941376412 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d00197820 con 0x7f6d001007e0 2024-09-10T03:39:39.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.577+0000 7f6cff7fe640 1 --2- 172.21.15.112:0/3941376412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d001007e0 0x7f6d0019d6a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000b420 tx=0x7f6cf000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:39.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.577+0000 7f6cfd7fa640 1 -- 172.21.15.112:0/3941376412 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6cf0011630 con 0x7f6d001007e0 2024-09-10T03:39:39.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.577+0000 7f6d05f38640 1 -- 172.21.15.112:0/3941376412 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d00197b10 con 0x7f6d001007e0 2024-09-10T03:39:39.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.577+0000 7f6cfd7fa640 1 -- 172.21.15.112:0/3941376412 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6cf0016070 con 0x7f6d001007e0 2024-09-10T03:39:39.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.577+0000 7f6cfd7fa640 1 -- 172.21.15.112:0/3941376412 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6cf0010d40 con 0x7f6d001007e0 2024-09-10T03:39:39.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.577+0000 7f6d05f38640 1 -- 172.21.15.112:0/3941376412 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d00198070 con 0x7f6d001007e0 2024-09-10T03:39:39.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.578+0000 7f6d05f38640 1 -- 172.21.15.112:0/3941376412 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d00101f20 con 0x7f6d001007e0 2024-09-10T03:39:39.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.579+0000 7f6cfd7fa640 1 -- 172.21.15.112:0/3941376412 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f6cf001b460 con 0x7f6d001007e0 2024-09-10T03:39:39.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.579+0000 7f6cfd7fa640 1 --2- 172.21.15.112:0/3941376412 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6cec06de30 0x7f6cec0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:39.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.579+0000 7f6cfd7fa640 1 -- 172.21.15.112:0/3941376412 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f6cf008d0b0 con 0x7f6d001007e0 2024-09-10T03:39:39.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.579+0000 7f6cf7fff640 1 --2- 172.21.15.112:0/3941376412 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6cec06de30 0x7f6cec0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:39.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.580+0000 7f6cf7fff640 1 --2- 172.21.15.112:0/3941376412 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6cec06de30 0x7f6cec0702f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6d00198b70 tx=0x7f6ce8031400 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:39.584 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.583+0000 7f6cfd7fa640 1 -- 172.21.15.112:0/3941376412 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6cf005af90 con 0x7f6d001007e0 2024-09-10T03:39:39.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.748+0000 7f6d05f38640 1 -- 172.21.15.112:0/3941376412 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6d00109d80 con 0x7f6d001007e0 2024-09-10T03:39:39.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.749+0000 7f6cfd7fa640 1 -- 172.21.15.112:0/3941376412 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) ==== 93+0+10 (secure 0 0 0) 0x7f6cf005ec40 con 0x7f6d001007e0 2024-09-10T03:39:39.752 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 1 2024-09-10T03:39:39.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.751+0000 7f6d05f38640 1 -- 172.21.15.112:0/3941376412 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6cec06de30 msgr2=0x7f6cec0702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:39.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.751+0000 7f6d05f38640 1 --2- 172.21.15.112:0/3941376412 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6cec06de30 0x7f6cec0702f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6d00198b70 tx=0x7f6ce8031400 comp rx=0 tx=0).stop 2024-09-10T03:39:39.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.751+0000 7f6d05f38640 1 -- 172.21.15.112:0/3941376412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d001007e0 msgr2=0x7f6d0019d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:39.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.751+0000 7f6d05f38640 1 --2- 172.21.15.112:0/3941376412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d001007e0 0x7f6d0019d6a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000b420 tx=0x7f6cf000b8e0 comp rx=0 tx=0).stop 2024-09-10T03:39:39.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.751+0000 7f6d05f38640 1 -- 172.21.15.112:0/3941376412 shutdown_connections 2024-09-10T03:39:39.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.751+0000 7f6d05f38640 1 --2- 172.21.15.112:0/3941376412 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6cec06de30 0x7f6cec0702f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:39.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.751+0000 7f6d05f38640 1 --2- 172.21.15.112:0/3941376412 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6d001067e0 0x7f6d0019dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:39.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.751+0000 7f6d05f38640 1 --2- 172.21.15.112:0/3941376412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d001007e0 0x7f6d0019d6a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:39.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.751+0000 7f6d05f38640 1 -- 172.21.15.112:0/3941376412 >> 172.21.15.112:0/3941376412 conn(0x7f6d000fc4c0 msgr2=0x7f6d00069e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:39.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.751+0000 7f6d05f38640 1 -- 172.21.15.112:0/3941376412 shutdown_connections 2024-09-10T03:39:39.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:39.751+0000 7f6d05f38640 1 -- 172.21.15.112:0/3941376412 wait complete. 2024-09-10T03:39:39.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:39 smithi112 ceph-mon[27737]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:40.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:39 smithi169 ceph-mon[35668]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:40.526 INFO:tasks.cephadm:Setting up client nodes... 2024-09-10T03:39:40.527 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-10T03:39:40.778 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:40.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:40 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3941376412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-10T03:39:41.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:40 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3941376412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-10T03:39:41.261 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.247+0000 7f14148af640 1 -- 172.21.15.112:0/2288786425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1410101a80 msgr2=0x7f1410101f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:41.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.247+0000 7f14148af640 1 --2- 172.21.15.112:0/2288786425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1410101a80 0x7f1410101f00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1404009880 tx=0x7f140402a4a0 comp rx=0 tx=0).stop 2024-09-10T03:39:41.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.247+0000 7f14148af640 1 -- 172.21.15.112:0/2288786425 shutdown_connections 2024-09-10T03:39:41.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.247+0000 7f14148af640 1 --2- 172.21.15.112:0/2288786425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1410101a80 0x7f1410101f00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:41.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.247+0000 7f14148af640 1 --2- 172.21.15.112:0/2288786425 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1410100880 0x7f1410100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:41.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.247+0000 7f14148af640 1 -- 172.21.15.112:0/2288786425 >> 172.21.15.112:0/2288786425 conn(0x7f14100fc010 msgr2=0x7f14100fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:41.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.247+0000 7f14148af640 1 -- 172.21.15.112:0/2288786425 shutdown_connections 2024-09-10T03:39:41.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.247+0000 7f14148af640 1 -- 172.21.15.112:0/2288786425 wait complete. 2024-09-10T03:39:41.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.248+0000 7f14148af640 1 Processor -- start 2024-09-10T03:39:41.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.248+0000 7f14148af640 1 -- start start 2024-09-10T03:39:41.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.248+0000 7f14148af640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1410100880 0x7f141006ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:41.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.248+0000 7f14148af640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1410101a80 0x7f141019c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:41.262 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.248+0000 7f14148af640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1410109740 con 0x7f1410100880 2024-09-10T03:39:41.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.248+0000 7f14148af640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f14101095c0 con 0x7f1410101a80 2024-09-10T03:39:41.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.248+0000 7f140e59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1410100880 0x7f141006ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:41.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.248+0000 7f140e59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1410100880 0x7f141006ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:44888/0 (socket says 172.21.15.112:44888) 2024-09-10T03:39:41.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.248+0000 7f140e59c640 1 -- 172.21.15.112:0/161970369 learned_addr learned my addr 172.21.15.112:0/161970369 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:41.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.248+0000 7f140dd9b640 1 --2- 172.21.15.112:0/161970369 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1410101a80 0x7f141019c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:41.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.248+0000 7f140e59c640 1 -- 172.21.15.112:0/161970369 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1410101a80 msgr2=0x7f141019c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:41.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.248+0000 7f140e59c640 1 --2- 172.21.15.112:0/161970369 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1410101a80 0x7f141019c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:41.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.248+0000 7f140e59c640 1 -- 172.21.15.112:0/161970369 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141019c880 con 0x7f1410100880 2024-09-10T03:39:41.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.248+0000 7f140dd9b640 1 --2- 172.21.15.112:0/161970369 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1410101a80 0x7f141019c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T03:39:41.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.249+0000 7f140e59c640 1 --2- 172.21.15.112:0/161970369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1410100880 0x7f141006ab30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f140000b3d0 tx=0x7f140000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:41.263 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.249+0000 7f13ff7fe640 1 -- 172.21.15.112:0/161970369 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f1400015020 con 0x7f1410100880 2024-09-10T03:39:41.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.249+0000 7f14148af640 1 -- 172.21.15.112:0/161970369 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f141019cb70 con 0x7f1410100880 2024-09-10T03:39:41.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.249+0000 7f13ff7fe640 1 -- 172.21.15.112:0/161970369 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1400011af0 con 0x7f1410100880 2024-09-10T03:39:41.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.249+0000 7f13ff7fe640 1 -- 172.21.15.112:0/161970369 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f1400010d90 con 0x7f1410100880 2024-09-10T03:39:41.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.249+0000 7f14148af640 1 -- 172.21.15.112:0/161970369 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f141019d100 con 0x7f1410100880 2024-09-10T03:39:41.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.250+0000 7f14148af640 1 -- 172.21.15.112:0/161970369 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1410100c80 con 0x7f1410100880 2024-09-10T03:39:41.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.250+0000 7f13ff7fe640 1 -- 172.21.15.112:0/161970369 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f1400018020 con 0x7f1410100880 2024-09-10T03:39:41.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.251+0000 7f13ff7fe640 1 --2- 172.21.15.112:0/161970369 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f13f806de30 0x7f13f80702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:41.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.251+0000 7f13ff7fe640 1 -- 172.21.15.112:0/161970369 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f140008d400 con 0x7f1410100880 2024-09-10T03:39:41.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.251+0000 7f140dd9b640 1 --2- 172.21.15.112:0/161970369 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f13f806de30 0x7f13f80702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:41.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.251+0000 7f140dd9b640 1 --2- 172.21.15.112:0/161970369 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f13f806de30 0x7f13f80702f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f14040096c0 tx=0x7f1404009650 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:41.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.255+0000 7f13ff7fe640 1 -- 172.21.15.112:0/161970369 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f140001f020 con 0x7f1410100880 2024-09-10T03:39:41.486 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.485+0000 7f14148af640 1 -- 172.21.15.112:0/161970369 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f1410062e40 con 0x7f1410100880 2024-09-10T03:39:41.488 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.487+0000 7f13ff7fe640 1 -- 172.21.15.112:0/161970369 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) ==== 170+0+59 (secure 0 0 0) 0x7f140005b250 con 0x7f1410100880 2024-09-10T03:39:41.488 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2024-09-10T03:39:41.488 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQB9v99mjBEGHRAAgN8AUAmXhDOP1KfXGyLMLg== 2024-09-10T03:39:41.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.490+0000 7f14148af640 1 -- 172.21.15.112:0/161970369 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f13f806de30 msgr2=0x7f13f80702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:41.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.490+0000 7f14148af640 1 --2- 172.21.15.112:0/161970369 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f13f806de30 0x7f13f80702f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f14040096c0 tx=0x7f1404009650 comp rx=0 tx=0).stop 2024-09-10T03:39:41.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.490+0000 7f14148af640 1 -- 172.21.15.112:0/161970369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1410100880 msgr2=0x7f141006ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:41.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.490+0000 7f14148af640 1 --2- 172.21.15.112:0/161970369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1410100880 0x7f141006ab30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f140000b3d0 tx=0x7f140000b890 comp rx=0 tx=0).stop 2024-09-10T03:39:41.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.490+0000 7f14148af640 1 -- 172.21.15.112:0/161970369 shutdown_connections 2024-09-10T03:39:41.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.490+0000 7f14148af640 1 --2- 172.21.15.112:0/161970369 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f13f806de30 0x7f13f80702f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:41.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.490+0000 7f14148af640 1 --2- 172.21.15.112:0/161970369 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1410101a80 0x7f141019c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:41.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.490+0000 7f14148af640 1 --2- 172.21.15.112:0/161970369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1410100880 0x7f141006ab30 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:41.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.490+0000 7f14148af640 1 -- 172.21.15.112:0/161970369 >> 172.21.15.112:0/161970369 conn(0x7f14100fc010 msgr2=0x7f1410104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:41.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.490+0000 7f14148af640 1 -- 172.21.15.112:0/161970369 shutdown_connections 2024-09-10T03:39:41.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:41.490+0000 7f14148af640 1 -- 172.21.15.112:0/161970369 wait complete. 2024-09-10T03:39:41.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:41 smithi112 ceph-mon[27737]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:41.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:41 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/161970369' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T03:39:41.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:41 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/161970369' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T03:39:42.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:41 smithi169 ceph-mon[35668]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:42.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:41 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/161970369' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T03:39:42.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:41 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/161970369' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T03:39:42.184 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T03:39:42.185 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-10T03:39:42.185 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-10T03:39:42.221 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-10T03:39:42.481 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi169/config 2024-09-10T03:39:42.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.936+0000 7f8078ef0640 1 -- 172.21.15.169:0/4159484484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8074100880 msgr2=0x7f8074100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:42.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.936+0000 7f8078ef0640 1 --2- 172.21.15.169:0/4159484484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8074100880 0x7f8074100c80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8064009880 tx=0x7f806402a4a0 comp rx=0 tx=0).stop 2024-09-10T03:39:42.940 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.936+0000 7f8078ef0640 1 -- 172.21.15.169:0/4159484484 shutdown_connections 2024-09-10T03:39:42.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.936+0000 7f8078ef0640 1 --2- 172.21.15.169:0/4159484484 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8074101a80 0x7f8074101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:42.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.936+0000 7f8078ef0640 1 --2- 172.21.15.169:0/4159484484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8074100880 0x7f8074100c80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:42.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.936+0000 7f8078ef0640 1 -- 172.21.15.169:0/4159484484 >> 172.21.15.169:0/4159484484 conn(0x7f80740fc010 msgr2=0x7f80740fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:42.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.936+0000 7f8078ef0640 1 -- 172.21.15.169:0/4159484484 shutdown_connections 2024-09-10T03:39:42.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.936+0000 7f8078ef0640 1 -- 172.21.15.169:0/4159484484 wait complete. 2024-09-10T03:39:42.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.937+0000 7f8078ef0640 1 Processor -- start 2024-09-10T03:39:42.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.937+0000 7f8078ef0640 1 -- start start 2024-09-10T03:39:42.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.937+0000 7f8078ef0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8074100880 0x7f807406a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:42.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.937+0000 7f8078ef0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8074101a80 0x7f807419c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:42.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.937+0000 7f8078ef0640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f80741096a0 con 0x7f8074100880 2024-09-10T03:39:42.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.937+0000 7f8078ef0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8074109520 con 0x7f8074101a80 2024-09-10T03:39:42.941 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.937+0000 7f8071d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8074101a80 0x7f807419c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:42.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.937+0000 7f8071d9b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8074101a80 0x7f807419c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:58394/0 (socket says 172.21.15.169:58394) 2024-09-10T03:39:42.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.937+0000 7f8071d9b640 1 -- 172.21.15.169:0/1450926372 learned_addr learned my addr 172.21.15.169:0/1450926372 (peer_addr_for_me v2:172.21.15.169:0/0) 2024-09-10T03:39:42.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.938+0000 7f807259c640 1 --2- 172.21.15.169:0/1450926372 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8074100880 0x7f807406a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:42.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.938+0000 7f8071d9b640 1 -- 172.21.15.169:0/1450926372 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8074100880 msgr2=0x7f807406a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:42.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.938+0000 7f8071d9b640 1 --2- 172.21.15.169:0/1450926372 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8074100880 0x7f807406a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:42.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.938+0000 7f8071d9b640 1 -- 172.21.15.169:0/1450926372 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f807419c7e0 con 0x7f8074101a80 2024-09-10T03:39:42.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.938+0000 7f807259c640 1 --2- 172.21.15.169:0/1450926372 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8074100880 0x7f807406a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T03:39:42.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.938+0000 7f8071d9b640 1 --2- 172.21.15.169:0/1450926372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8074101a80 0x7f807419c220 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f806800d900 tx=0x7f806800ddc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:42.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.938+0000 7f80637fe640 1 -- 172.21.15.169:0/1450926372 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8068017070 con 0x7f8074101a80 2024-09-10T03:39:42.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.938+0000 7f8078ef0640 1 -- 172.21.15.169:0/1450926372 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f807419cad0 con 0x7f8074101a80 2024-09-10T03:39:42.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.938+0000 7f80637fe640 1 -- 172.21.15.169:0/1450926372 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8068010460 con 0x7f8074101a80 2024-09-10T03:39:42.942 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.938+0000 7f80637fe640 1 -- 172.21.15.169:0/1450926372 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f806800f5f0 con 0x7f8074101a80 2024-09-10T03:39:42.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.938+0000 7f8078ef0640 1 -- 172.21.15.169:0/1450926372 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f807419d060 con 0x7f8074101a80 2024-09-10T03:39:42.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.939+0000 7f8078ef0640 1 -- 172.21.15.169:0/1450926372 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8074100c80 con 0x7f8074101a80 2024-09-10T03:39:42.943 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.939+0000 7f80637fe640 1 -- 172.21.15.169:0/1450926372 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f8068009a30 con 0x7f8074101a80 2024-09-10T03:39:42.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.940+0000 7f80637fe640 1 --2- 172.21.15.169:0/1450926372 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f805c06de30 0x7f805c0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:42.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.940+0000 7f80637fe640 1 -- 172.21.15.169:0/1450926372 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f806808d770 con 0x7f8074101a80 2024-09-10T03:39:42.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.940+0000 7f807259c640 1 --2- 172.21.15.169:0/1450926372 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f805c06de30 0x7f805c0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:42.944 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.941+0000 7f807259c640 1 --2- 172.21.15.169:0/1450926372 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f805c06de30 0x7f805c0702f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f80640096c0 tx=0x7f8064009650 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:42.946 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:42.945+0000 7f80637fe640 1 -- 172.21.15.169:0/1450926372 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f806805b5c0 con 0x7f8074101a80 2024-09-10T03:39:43.179 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:43.178+0000 7f8078ef0640 1 -- 172.21.15.169:0/1450926372 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f80741904b0 con 0x7f8074101a80 2024-09-10T03:39:43.184 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:43.182+0000 7f80637fe640 1 -- 172.21.15.169:0/1450926372 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) ==== 170+0+59 (secure 0 0 0) 0x7f806805f270 con 0x7f8074101a80 2024-09-10T03:39:43.184 INFO:teuthology.orchestra.run.smithi169.stdout:[client.1] 2024-09-10T03:39:43.184 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQB/v99mOc7CChAAGR3hHUXxDNFw4hnF2a6c4g== 2024-09-10T03:39:43.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:43.185+0000 7f8078ef0640 1 -- 172.21.15.169:0/1450926372 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f805c06de30 msgr2=0x7f805c0702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:43.186 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:43.185+0000 7f8078ef0640 1 --2- 172.21.15.169:0/1450926372 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f805c06de30 0x7f805c0702f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f80640096c0 tx=0x7f8064009650 comp rx=0 tx=0).stop 2024-09-10T03:39:43.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:43.185+0000 7f8078ef0640 1 -- 172.21.15.169:0/1450926372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8074101a80 msgr2=0x7f807419c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:43.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:43.185+0000 7f8078ef0640 1 --2- 172.21.15.169:0/1450926372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8074101a80 0x7f807419c220 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f806800d900 tx=0x7f806800ddc0 comp rx=0 tx=0).stop 2024-09-10T03:39:43.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:43.185+0000 7f8078ef0640 1 -- 172.21.15.169:0/1450926372 shutdown_connections 2024-09-10T03:39:43.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:43.185+0000 7f8078ef0640 1 --2- 172.21.15.169:0/1450926372 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f805c06de30 0x7f805c0702f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:43.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:43.185+0000 7f8078ef0640 1 --2- 172.21.15.169:0/1450926372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8074101a80 0x7f807419c220 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:43.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:43.185+0000 7f8078ef0640 1 --2- 172.21.15.169:0/1450926372 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8074100880 0x7f807406a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:43.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:43.185+0000 7f8078ef0640 1 -- 172.21.15.169:0/1450926372 >> 172.21.15.169:0/1450926372 conn(0x7f80740fc010 msgr2=0x7f8074104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:43.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:43.185+0000 7f8078ef0640 1 -- 172.21.15.169:0/1450926372 shutdown_connections 2024-09-10T03:39:43.187 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:39:43.185+0000 7f8078ef0640 1 -- 172.21.15.169:0/1450926372 wait complete. 2024-09-10T03:39:43.828 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-10T03:39:43.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-10T03:39:43.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-10T03:39:43.866 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph config log 1 --format=json 2024-09-10T03:39:44.123 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:44.244 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:43 smithi112 ceph-mon[27737]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:44.244 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:43 smithi112 ceph-mon[27737]: from='client.? 172.21.15.169:0/1450926372' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T03:39:44.244 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:43 smithi112 ceph-mon[27737]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T03:39:44.244 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:43 smithi112 ceph-mon[27737]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T03:39:44.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:43 smithi169 ceph-mon[35668]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:44.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:43 smithi169 ceph-mon[35668]: from='client.? 172.21.15.169:0/1450926372' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T03:39:44.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:43 smithi169 ceph-mon[35668]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T03:39:44.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:43 smithi169 ceph-mon[35668]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T03:39:44.577 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.574+0000 7f5893536640 1 -- 172.21.15.112:0/709305151 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f588c100810 msgr2=0x7f588c100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:44.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.574+0000 7f5893536640 1 --2- 172.21.15.112:0/709305151 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f588c100810 0x7f588c100c70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5880009c20 tx=0x7f588002a970 comp rx=0 tx=0).stop 2024-09-10T03:39:44.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.574+0000 7f5893536640 1 -- 172.21.15.112:0/709305151 shutdown_connections 2024-09-10T03:39:44.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.574+0000 7f5893536640 1 --2- 172.21.15.112:0/709305151 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f588c100810 0x7f588c100c70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:44.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.574+0000 7f5893536640 1 --2- 172.21.15.112:0/709305151 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f588c106810 0x7f588c106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:44.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.574+0000 7f5893536640 1 -- 172.21.15.112:0/709305151 >> 172.21.15.112:0/709305151 conn(0x7f588c0fc510 msgr2=0x7f588c0fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:44.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.574+0000 7f5893536640 1 -- 172.21.15.112:0/709305151 shutdown_connections 2024-09-10T03:39:44.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.574+0000 7f5893536640 1 -- 172.21.15.112:0/709305151 wait complete. 2024-09-10T03:39:44.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.574+0000 7f5893536640 1 Processor -- start 2024-09-10T03:39:44.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.574+0000 7f5893536640 1 -- start start 2024-09-10T03:39:44.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.575+0000 7f5893536640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f588c100810 0x7f588c19c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:44.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.575+0000 7f5893536640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f588c106810 0x7f588c19cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:44.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.575+0000 7f5893536640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f588c10dce0 con 0x7f588c106810 2024-09-10T03:39:44.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.575+0000 7f5893536640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f588c10db60 con 0x7f588c100810 2024-09-10T03:39:44.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.575+0000 7f5890ad1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f588c106810 0x7f588c19cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:44.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.575+0000 7f5890ad1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f588c106810 0x7f588c19cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:44894/0 (socket says 172.21.15.112:44894) 2024-09-10T03:39:44.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.575+0000 7f5890ad1640 1 -- 172.21.15.112:0/3682842261 learned_addr learned my addr 172.21.15.112:0/3682842261 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:44.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.575+0000 7f58912d2640 1 --2- 172.21.15.112:0/3682842261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f588c100810 0x7f588c19c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:44.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.575+0000 7f5890ad1640 1 -- 172.21.15.112:0/3682842261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f588c100810 msgr2=0x7f588c19c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:44.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.575+0000 7f5890ad1640 1 --2- 172.21.15.112:0/3682842261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f588c100810 0x7f588c19c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:44.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.575+0000 7f5890ad1640 1 -- 172.21.15.112:0/3682842261 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f588c1966f0 con 0x7f588c106810 2024-09-10T03:39:44.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.575+0000 7f58912d2640 1 --2- 172.21.15.112:0/3682842261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f588c100810 0x7f588c19c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T03:39:44.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.575+0000 7f5890ad1640 1 --2- 172.21.15.112:0/3682842261 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f588c106810 0x7f588c19cb60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f588002ae90 tx=0x7f588002fd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:44.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.575+0000 7f588a7fc640 1 -- 172.21.15.112:0/3682842261 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f5880047070 con 0x7f588c106810 2024-09-10T03:39:44.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.575+0000 7f588a7fc640 1 -- 172.21.15.112:0/3682842261 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5880003e30 con 0x7f588c106810 2024-09-10T03:39:44.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.575+0000 7f5893536640 1 -- 172.21.15.112:0/3682842261 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f588c196980 con 0x7f588c106810 2024-09-10T03:39:44.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.575+0000 7f588a7fc640 1 -- 172.21.15.112:0/3682842261 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f5880034070 con 0x7f588c106810 2024-09-10T03:39:44.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.576+0000 7f5893536640 1 -- 172.21.15.112:0/3682842261 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f588c196e80 con 0x7f588c106810 2024-09-10T03:39:44.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.576+0000 7f5893536640 1 -- 172.21.15.112:0/3682842261 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f588c101f50 con 0x7f588c106810 2024-09-10T03:39:44.580 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.577+0000 7f588a7fc640 1 -- 172.21.15.112:0/3682842261 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f588004a070 con 0x7f588c106810 2024-09-10T03:39:44.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.578+0000 7f588a7fc640 1 --2- 172.21.15.112:0/3682842261 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f587406de90 0x7f5874070350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:44.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.578+0000 7f588a7fc640 1 -- 172.21.15.112:0/3682842261 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f58800ae420 con 0x7f588c106810 2024-09-10T03:39:44.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.578+0000 7f58912d2640 1 --2- 172.21.15.112:0/3682842261 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f587406de90 0x7f5874070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:44.581 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.578+0000 7f58912d2640 1 --2- 172.21.15.112:0/3682842261 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f587406de90 0x7f5874070350 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f587c009b50 tx=0x7f587c005f90 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:44.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.582+0000 7f588a7fc640 1 -- 172.21.15.112:0/3682842261 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f588003f070 con 0x7f588c106810 2024-09-10T03:39:44.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.748+0000 7f5893536640 1 -- 172.21.15.112:0/3682842261 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f588c108d80 con 0x7f588c106810 2024-09-10T03:39:44.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.749+0000 7f588a7fc640 1 -- 172.21.15.112:0/3682842261 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v41) ==== 86+0+160 (secure 0 0 0) 0x7f588007c380 con 0x7f588c106810 2024-09-10T03:39:44.750 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:39:44.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.751+0000 7f5893536640 1 -- 172.21.15.112:0/3682842261 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f587406de90 msgr2=0x7f5874070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:44.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.751+0000 7f5893536640 1 --2- 172.21.15.112:0/3682842261 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f587406de90 0x7f5874070350 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f587c009b50 tx=0x7f587c005f90 comp rx=0 tx=0).stop 2024-09-10T03:39:44.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.751+0000 7f5893536640 1 -- 172.21.15.112:0/3682842261 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f588c106810 msgr2=0x7f588c19cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:44.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.751+0000 7f5893536640 1 --2- 172.21.15.112:0/3682842261 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f588c106810 0x7f588c19cb60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f588002ae90 tx=0x7f588002fd60 comp rx=0 tx=0).stop 2024-09-10T03:39:44.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.751+0000 7f5893536640 1 -- 172.21.15.112:0/3682842261 shutdown_connections 2024-09-10T03:39:44.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.751+0000 7f5893536640 1 --2- 172.21.15.112:0/3682842261 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f587406de90 0x7f5874070350 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:44.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.751+0000 7f5893536640 1 --2- 172.21.15.112:0/3682842261 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f588c106810 0x7f588c19cb60 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:44.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.751+0000 7f5893536640 1 --2- 172.21.15.112:0/3682842261 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f588c100810 0x7f588c19c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:44.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.751+0000 7f5893536640 1 -- 172.21.15.112:0/3682842261 >> 172.21.15.112:0/3682842261 conn(0x7f588c0fc510 msgr2=0x7f588c0fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:44.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.751+0000 7f5893536640 1 -- 172.21.15.112:0/3682842261 shutdown_connections 2024-09-10T03:39:44.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:44.751+0000 7f5893536640 1 -- 172.21.15.112:0/3682842261 wait complete. 2024-09-10T03:39:45.031 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:44 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3682842261' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-10T03:39:45.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:44 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3682842261' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-10T03:39:45.443 INFO:teuthology.orchestra.run.smithi112.stdout:[{"version":41,"timestamp":"2024-09-10T03:39:34.365705+0000","name":"","changes":[{"name":"osd.5/osd_mclock_max_capacity_iops_ssd","new_value":"8393.324410"}]}] 2024-09-10T03:39:45.444 INFO:tasks.ceph_manager:config epoch is 41 2024-09-10T03:39:45.444 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-10T03:39:45.444 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-10T03:39:45.444 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph mgr dump --format=json 2024-09-10T03:39:45.700 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:46.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:45 smithi112 ceph-mon[27737]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:46.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.133+0000 7fc2a20cd640 1 -- 172.21.15.112:0/3112095192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc29c101930 msgr2=0x7fc29c101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:46.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.133+0000 7fc2a20cd640 1 --2- 172.21.15.112:0/3112095192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc29c101930 0x7fc29c101d90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc290009880 tx=0x7fc29002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:39:46.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.133+0000 7fc2a20cd640 1 -- 172.21.15.112:0/3112095192 shutdown_connections 2024-09-10T03:39:46.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.133+0000 7fc2a20cd640 1 --2- 172.21.15.112:0/3112095192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc29c101930 0x7fc29c101d90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:46.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.133+0000 7fc2a20cd640 1 --2- 172.21.15.112:0/3112095192 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc29c107930 0x7fc29c107d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:46.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.133+0000 7fc2a20cd640 1 -- 172.21.15.112:0/3112095192 >> 172.21.15.112:0/3112095192 conn(0x7fc29c0fd630 msgr2=0x7fc29c0ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:46.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.133+0000 7fc2a20cd640 1 -- 172.21.15.112:0/3112095192 shutdown_connections 2024-09-10T03:39:46.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.133+0000 7fc2a20cd640 1 -- 172.21.15.112:0/3112095192 wait complete. 2024-09-10T03:39:46.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.134+0000 7fc2a20cd640 1 Processor -- start 2024-09-10T03:39:46.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.134+0000 7fc2a20cd640 1 -- start start 2024-09-10T03:39:46.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.134+0000 7fc2a20cd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc29c101930 0x7fc29c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:46.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.134+0000 7fc2a20cd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc29c107930 0x7fc29c074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:46.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.134+0000 7fc2a20cd640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc29c10ee60 con 0x7fc29c107930 2024-09-10T03:39:46.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.134+0000 7fc2a20cd640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc29c10ece0 con 0x7fc29c101930 2024-09-10T03:39:46.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.134+0000 7fc29affd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc29c107930 0x7fc29c074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:46.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.134+0000 7fc29affd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc29c107930 0x7fc29c074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:42100/0 (socket says 172.21.15.112:42100) 2024-09-10T03:39:46.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.134+0000 7fc29affd640 1 -- 172.21.15.112:0/3034602141 learned_addr learned my addr 172.21.15.112:0/3034602141 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:46.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.134+0000 7fc29b7fe640 1 --2- 172.21.15.112:0/3034602141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc29c101930 0x7fc29c0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:46.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.135+0000 7fc29affd640 1 -- 172.21.15.112:0/3034602141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc29c101930 msgr2=0x7fc29c0744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:46.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.135+0000 7fc29affd640 1 --2- 172.21.15.112:0/3034602141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc29c101930 0x7fc29c0744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:46.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.135+0000 7fc29affd640 1 -- 172.21.15.112:0/3034602141 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc29c074f70 con 0x7fc29c107930 2024-09-10T03:39:46.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.135+0000 7fc29affd640 1 --2- 172.21.15.112:0/3034602141 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc29c107930 0x7fc29c074a30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc290031460 tx=0x7fc2900048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:46.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.135+0000 7fc298ff9640 1 -- 172.21.15.112:0/3034602141 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fc29003f070 con 0x7fc29c107930 2024-09-10T03:39:46.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.135+0000 7fc2a20cd640 1 -- 172.21.15.112:0/3034602141 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc29c075200 con 0x7fc29c107930 2024-09-10T03:39:46.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.135+0000 7fc298ff9640 1 -- 172.21.15.112:0/3034602141 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc290004cf0 con 0x7fc29c107930 2024-09-10T03:39:46.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.135+0000 7fc298ff9640 1 -- 172.21.15.112:0/3034602141 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fc290035030 con 0x7fc29c107930 2024-09-10T03:39:46.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.135+0000 7fc2a20cd640 1 -- 172.21.15.112:0/3034602141 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc29c1a3e30 con 0x7fc29c107930 2024-09-10T03:39:46.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.136+0000 7fc2a20cd640 1 -- 172.21.15.112:0/3034602141 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc29c103070 con 0x7fc29c107930 2024-09-10T03:39:46.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.137+0000 7fc298ff9640 1 -- 172.21.15.112:0/3034602141 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fc290042070 con 0x7fc29c107930 2024-09-10T03:39:46.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.137+0000 7fc298ff9640 1 --2- 172.21.15.112:0/3034602141 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fc28406de30 0x7fc2840702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:46.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.137+0000 7fc298ff9640 1 -- 172.21.15.112:0/3034602141 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7fc2900ae280 con 0x7fc29c107930 2024-09-10T03:39:46.163 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.137+0000 7fc29b7fe640 1 --2- 172.21.15.112:0/3034602141 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fc28406de30 0x7fc2840702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:46.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.138+0000 7fc29b7fe640 1 --2- 172.21.15.112:0/3034602141 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fc28406de30 0x7fc2840702f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc28c009990 tx=0x7fc28c009110 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:46.164 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.141+0000 7fc298ff9640 1 -- 172.21.15.112:0/3034602141 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc29007c160 con 0x7fc29c107930 2024-09-10T03:39:46.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:45 smithi169 ceph-mon[35668]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:46.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.358+0000 7fc2a20cd640 1 -- 172.21.15.112:0/3034602141 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fc29c075a20 con 0x7fc29c107930 2024-09-10T03:39:46.362 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.361+0000 7fc298ff9640 1 -- 172.21.15.112:0/3034602141 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) ==== 74+0+176009 (secure 0 0 0) 0x7fc29007fe10 con 0x7fc29c107930 2024-09-10T03:39:46.364 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:39:46.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.363+0000 7fc2a20cd640 1 -- 172.21.15.112:0/3034602141 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fc28406de30 msgr2=0x7fc2840702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:46.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.363+0000 7fc2a20cd640 1 --2- 172.21.15.112:0/3034602141 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fc28406de30 0x7fc2840702f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc28c009990 tx=0x7fc28c009110 comp rx=0 tx=0).stop 2024-09-10T03:39:46.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.363+0000 7fc2a20cd640 1 -- 172.21.15.112:0/3034602141 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc29c107930 msgr2=0x7fc29c074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:46.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.363+0000 7fc2a20cd640 1 --2- 172.21.15.112:0/3034602141 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc29c107930 0x7fc29c074a30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc290031460 tx=0x7fc2900048f0 comp rx=0 tx=0).stop 2024-09-10T03:39:46.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.363+0000 7fc2a20cd640 1 -- 172.21.15.112:0/3034602141 shutdown_connections 2024-09-10T03:39:46.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.363+0000 7fc2a20cd640 1 --2- 172.21.15.112:0/3034602141 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fc28406de30 0x7fc2840702f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:46.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.364+0000 7fc2a20cd640 1 --2- 172.21.15.112:0/3034602141 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc29c107930 0x7fc29c074a30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:46.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.364+0000 7fc2a20cd640 1 --2- 172.21.15.112:0/3034602141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc29c101930 0x7fc29c0744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:46.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.364+0000 7fc2a20cd640 1 -- 172.21.15.112:0/3034602141 >> 172.21.15.112:0/3034602141 conn(0x7fc29c0fd630 msgr2=0x7fc29c0fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:46.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.364+0000 7fc2a20cd640 1 -- 172.21.15.112:0/3034602141 shutdown_connections 2024-09-10T03:39:46.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:46.364+0000 7fc2a20cd640 1 -- 172.21.15.112:0/3034602141 wait complete. 2024-09-10T03:39:47.050 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":19,"flags":0,"active_gid":14221,"active_name":"smithi112.pqrlrb","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3600199018},{"type":"v1","addr":"172.21.15.112:6801","nonce":3600199018}]},"active_addr":"172.21.15.112:6801/3600199018","active_change":"2024-09-10T03:35:34.814214+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":14246,"name":"smithi169.bkwgjl","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.112:8443/","prometheus":"http://172.21.15.112:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":2493682922}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":3131050348}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":3783280263}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":1156205439}]}]} 2024-09-10T03:39:47.057 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-10T03:39:47.057 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-10T03:39:47.058 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T03:39:47.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:46 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3034602141' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-10T03:39:47.304 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:47.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:46 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3034602141' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-10T03:39:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.746+0000 7f203325a640 1 -- 172.21.15.112:0/2388893655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f202c1067e0 msgr2=0x7f202c106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.746+0000 7f203325a640 1 --2- 172.21.15.112:0/2388893655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f202c1067e0 0x7f202c106bc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2020009880 tx=0x7f202002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:39:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.746+0000 7f203325a640 1 -- 172.21.15.112:0/2388893655 shutdown_connections 2024-09-10T03:39:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.746+0000 7f203325a640 1 --2- 172.21.15.112:0/2388893655 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f202c1007e0 0x7f202c100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.746+0000 7f203325a640 1 --2- 172.21.15.112:0/2388893655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f202c1067e0 0x7f202c106bc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.746+0000 7f203325a640 1 -- 172.21.15.112:0/2388893655 >> 172.21.15.112:0/2388893655 conn(0x7f202c0fc4c0 msgr2=0x7f202c0fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.746+0000 7f203325a640 1 -- 172.21.15.112:0/2388893655 shutdown_connections 2024-09-10T03:39:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.746+0000 7f203325a640 1 -- 172.21.15.112:0/2388893655 wait complete. 2024-09-10T03:39:47.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.747+0000 7f203325a640 1 Processor -- start 2024-09-10T03:39:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.747+0000 7f203325a640 1 -- start start 2024-09-10T03:39:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.747+0000 7f203325a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f202c1007e0 0x7f202c19d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.747+0000 7f203325a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f202c1067e0 0x7f202c19dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.747+0000 7f203325a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f202c10dcb0 con 0x7f202c1067e0 2024-09-10T03:39:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.747+0000 7f203325a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f202c10db30 con 0x7f202c1007e0 2024-09-10T03:39:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.747+0000 7f202bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f202c1067e0 0x7f202c19dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.747+0000 7f202bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f202c1067e0 0x7f202c19dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:42118/0 (socket says 172.21.15.112:42118) 2024-09-10T03:39:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.747+0000 7f202bfff640 1 -- 172.21.15.112:0/2105124960 learned_addr learned my addr 172.21.15.112:0/2105124960 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.747+0000 7f2030ff6640 1 --2- 172.21.15.112:0/2105124960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f202c1007e0 0x7f202c19d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.748+0000 7f202bfff640 1 -- 172.21.15.112:0/2105124960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f202c1007e0 msgr2=0x7f202c19d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.748+0000 7f202bfff640 1 --2- 172.21.15.112:0/2105124960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f202c1007e0 0x7f202c19d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.748+0000 7f202bfff640 1 -- 172.21.15.112:0/2105124960 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202c1977e0 con 0x7f202c1067e0 2024-09-10T03:39:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.748+0000 7f202bfff640 1 --2- 172.21.15.112:0/2105124960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f202c1067e0 0x7f202c19dba0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f201800d6f0 tx=0x7f201800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:47.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.748+0000 7f2029ffb640 1 -- 172.21.15.112:0/2105124960 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2018017070 con 0x7f202c1067e0 2024-09-10T03:39:47.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.748+0000 7f2029ffb640 1 -- 172.21.15.112:0/2105124960 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f201800de00 con 0x7f202c1067e0 2024-09-10T03:39:47.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.748+0000 7f203325a640 1 -- 172.21.15.112:0/2105124960 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f202c197ad0 con 0x7f202c1067e0 2024-09-10T03:39:47.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.748+0000 7f2029ffb640 1 -- 172.21.15.112:0/2105124960 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2018004bb0 con 0x7f202c1067e0 2024-09-10T03:39:47.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.748+0000 7f203325a640 1 -- 172.21.15.112:0/2105124960 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f202c198060 con 0x7f202c1067e0 2024-09-10T03:39:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.749+0000 7f203325a640 1 -- 172.21.15.112:0/2105124960 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f202c101f20 con 0x7f202c1067e0 2024-09-10T03:39:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.750+0000 7f2029ffb640 1 -- 172.21.15.112:0/2105124960 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f2018004e60 con 0x7f202c1067e0 2024-09-10T03:39:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.750+0000 7f2029ffb640 1 --2- 172.21.15.112:0/2105124960 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f201c06de30 0x7f201c0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.750+0000 7f2030ff6640 1 --2- 172.21.15.112:0/2105124960 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f201c06de30 0x7f201c0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.750+0000 7f2029ffb640 1 -- 172.21.15.112:0/2105124960 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f201808d390 con 0x7f202c1067e0 2024-09-10T03:39:47.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.751+0000 7f2030ff6640 1 --2- 172.21.15.112:0/2105124960 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f201c06de30 0x7f201c0702f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f20200096c0 tx=0x7f2020031400 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:47.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.755+0000 7f2029ffb640 1 -- 172.21.15.112:0/2105124960 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f201805ef20 con 0x7f202c1067e0 2024-09-10T03:39:47.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.921+0000 7f203325a640 1 -- 172.21.15.112:0/2105124960 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f202c198310 con 0x7f202c1067e0 2024-09-10T03:39:47.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.921+0000 7f2029ffb640 1 -- 172.21.15.112:0/2105124960 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+11420 (secure 0 0 0) 0x7f2018063750 con 0x7f202c1067e0 2024-09-10T03:39:47.922 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:39:47.922 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":39,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","created":"2024-09-10T03:33:31.106241+0000","modified":"2024-09-10T03:39:36.567053+0000","last_up_change":"2024-09-10T03:39:35.366928+0000","last_in_change":"2024-09-10T03:39:11.309435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T03:38:15.123477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ae9a7fcc-9d61-4914-b88f-da4fc76d5b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":378527782},{"type":"v1","addr":"172.21.15.112:6803","nonce":378527782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":378527782},{"type":"v1","addr":"172.21.15.112:6805","nonce":378527782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":378527782},{"type":"v1","addr":"172.21.15.112:6809","nonce":378527782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":378527782},{"type":"v1","addr":"172.21.15.112:6807","nonce":378527782}]},"public_addr":"172.21.15.112:6803/378527782","cluster_addr":"172.21.15.112:6805/378527782","heartbeat_back_addr":"172.21.15.112:6809/378527782","heartbeat_front_addr":"172.21.15.112:6807/378527782","state":["exists","up"]},{"osd":1,"uuid":"41b3eaeb-7456-42dd-9405-0770a7129bc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":305082264},{"type":"v1","addr":"172.21.15.112:6811","nonce":305082264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":305082264},{"type":"v1","addr":"172.21.15.112:6813","nonce":305082264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":305082264},{"type":"v1","addr":"172.21.15.112:6817","nonce":305082264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":305082264},{"type":"v1","addr":"172.21.15.112:6815","nonce":305082264}]},"public_addr":"172.21.15.112:6811/305082264","cluster_addr":"172.21.15.112:6813/305082264","heartbeat_back_addr":"172.21.15.112:6817/305082264","heartbeat_front_addr":"172.21.15.112:6815/305082264","state":["exists","up"]},{"osd":2,"uuid":"e7d8e980-cf86-473c-805c-61ebb62e4d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3325040968},{"type":"v1","addr":"172.21.15.112:6819","nonce":3325040968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3325040968},{"type":"v1","addr":"172.21.15.112:6821","nonce":3325040968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3325040968},{"type":"v1","addr":"172.21.15.112:6825","nonce":3325040968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3325040968},{"type":"v1","addr":"172.21.15.112:6823","nonce":3325040968}]},"public_addr":"172.21.15.112:6819/3325040968","cluster_addr":"172.21.15.112:6821/3325040968","heartbeat_back_addr":"172.21.15.112:6825/3325040968","heartbeat_front_addr":"172.21.15.112:6823/3325040968","state":["exists","up"]},{"osd":3,"uuid":"eec71a12-c779-49af-a994-414f77d0ec78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3830555794},{"type":"v1","addr":"172.21.15.169:6801","nonce":3830555794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3830555794},{"type":"v1","addr":"172.21.15.169:6803","nonce":3830555794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":3830555794},{"type":"v1","addr":"172.21.15.169:6807","nonce":3830555794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":3830555794},{"type":"v1","addr":"172.21.15.169:6805","nonce":3830555794}]},"public_addr":"172.21.15.169:6801/3830555794","cluster_addr":"172.21.15.169:6803/3830555794","heartbeat_back_addr":"172.21.15.169:6807/3830555794","heartbeat_front_addr":"172.21.15.169:6805/3830555794","state":["exists","up"]},{"osd":4,"uuid":"77c831ef-a06d-4f4f-ad2a-c2cb04613fe0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":2285032153},{"type":"v1","addr":"172.21.15.169:6809","nonce":2285032153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":2285032153},{"type":"v1","addr":"172.21.15.169:6811","nonce":2285032153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":2285032153},{"type":"v1","addr":"172.21.15.169:6815","nonce":2285032153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":2285032153},{"type":"v1","addr":"172.21.15.169:6813","nonce":2285032153}]},"public_addr":"172.21.15.169:6809/2285032153","cluster_addr":"172.21.15.169:6811/2285032153","heartbeat_back_addr":"172.21.15.169:6815/2285032153","heartbeat_front_addr":"172.21.15.169:6813/2285032153","state":["exists","up"]},{"osd":5,"uuid":"346b5969-a50d-4340-a839-ef1085e248fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":893753355},{"type":"v1","addr":"172.21.15.169:6817","nonce":893753355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":893753355},{"type":"v1","addr":"172.21.15.169:6819","nonce":893753355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":893753355},{"type":"v1","addr":"172.21.15.169:6823","nonce":893753355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":893753355},{"type":"v1","addr":"172.21.15.169:6821","nonce":893753355}]},"public_addr":"172.21.15.169:6817/893753355","cluster_addr":"172.21.15.169:6819/893753355","heartbeat_back_addr":"172.21.15.169:6823/893753355","heartbeat_front_addr":"172.21.15.169:6821/893753355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:37:09.717391+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:37:43.057817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:38:12.048486+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:38:37.732503+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:39:06.199278+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:39:32.899049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.112:6801/171256542":"2024-09-11T03:35:34.814063+0000","172.21.15.112:0/3104768277":"2024-09-11T03:34:03.009905+0000","172.21.15.112:6800/171256542":"2024-09-11T03:35:34.814063+0000","172.21.15.112:6800/3423408536":"2024-09-11T03:34:03.009905+0000","172.21.15.112:0/3329867731":"2024-09-11T03:34:03.009905+0000","172.21.15.112:0/3444044994":"2024-09-11T03:34:03.009905+0000","172.21.15.112:0/288016527":"2024-09-11T03:34:33.723096+0000","172.21.15.112:0/3338248392":"2024-09-11T03:34:33.723096+0000","172.21.15.112:6801/4183006970":"2024-09-11T03:34:33.723096+0000","172.21.15.112:0/4078185439":"2024-09-11T03:34:33.723096+0000","172.21.15.112:6800/4183006970":"2024-09-11T03:34:33.723096+0000","172.21.15.112:0/3410252058":"2024-09-11T03:35:34.814063+0000","172.21.15.112:6801/3423408536":"2024-09-11T03:34:03.009905+0000","172.21.15.112:0/1394011979":"2024-09-11T03:35:34.814063+0000","172.21.15.112:0/522295971":"2024-09-11T03:35:34.814063+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T03:39:47.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.924+0000 7f203325a640 1 -- 172.21.15.112:0/2105124960 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f201c06de30 msgr2=0x7f201c0702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:47.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.924+0000 7f203325a640 1 --2- 172.21.15.112:0/2105124960 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f201c06de30 0x7f201c0702f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f20200096c0 tx=0x7f2020031400 comp rx=0 tx=0).stop 2024-09-10T03:39:47.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.924+0000 7f203325a640 1 -- 172.21.15.112:0/2105124960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f202c1067e0 msgr2=0x7f202c19dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:47.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.924+0000 7f203325a640 1 --2- 172.21.15.112:0/2105124960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f202c1067e0 0x7f202c19dba0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f201800d6f0 tx=0x7f201800dbb0 comp rx=0 tx=0).stop 2024-09-10T03:39:47.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.924+0000 7f203325a640 1 -- 172.21.15.112:0/2105124960 shutdown_connections 2024-09-10T03:39:47.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.924+0000 7f203325a640 1 --2- 172.21.15.112:0/2105124960 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f201c06de30 0x7f201c0702f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:47.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.924+0000 7f203325a640 1 --2- 172.21.15.112:0/2105124960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f202c1067e0 0x7f202c19dba0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:47.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.924+0000 7f203325a640 1 --2- 172.21.15.112:0/2105124960 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f202c1007e0 0x7f202c19d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:47.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.924+0000 7f203325a640 1 -- 172.21.15.112:0/2105124960 >> 172.21.15.112:0/2105124960 conn(0x7f202c0fc4c0 msgr2=0x7f202c104630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:47.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.924+0000 7f203325a640 1 -- 172.21.15.112:0/2105124960 shutdown_connections 2024-09-10T03:39:47.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:47.924+0000 7f203325a640 1 -- 172.21.15.112:0/2105124960 wait complete. 2024-09-10T03:39:48.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:47 smithi112 ceph-mon[27737]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:48.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:47 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2105124960' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T03:39:48.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:47 smithi169 ceph-mon[35668]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:48.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:47 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/2105124960' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T03:39:48.610 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-10T03:39:48.611 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T03:39:48.859 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:49.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.338+0000 7f02168ec640 1 -- 172.21.15.112:0/4135108488 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0210107930 msgr2=0x7f0210107d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:49.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.338+0000 7f02168ec640 1 --2- 172.21.15.112:0/4135108488 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0210107930 0x7f0210107d10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0200009880 tx=0x7f020002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:39:49.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.338+0000 7f02168ec640 1 -- 172.21.15.112:0/4135108488 shutdown_connections 2024-09-10T03:39:49.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.338+0000 7f02168ec640 1 --2- 172.21.15.112:0/4135108488 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0210101930 0x7f0210101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:49.368 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.338+0000 7f02168ec640 1 --2- 172.21.15.112:0/4135108488 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0210107930 0x7f0210107d10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:49.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.338+0000 7f02168ec640 1 -- 172.21.15.112:0/4135108488 >> 172.21.15.112:0/4135108488 conn(0x7f02100fd630 msgr2=0x7f02100ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:49.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.338+0000 7f02168ec640 1 -- 172.21.15.112:0/4135108488 shutdown_connections 2024-09-10T03:39:49.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.338+0000 7f02168ec640 1 -- 172.21.15.112:0/4135108488 wait complete. 2024-09-10T03:39:49.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.339+0000 7f02168ec640 1 Processor -- start 2024-09-10T03:39:49.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.339+0000 7f02168ec640 1 -- start start 2024-09-10T03:39:49.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.339+0000 7f02168ec640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0210101930 0x7f02100744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:49.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.339+0000 7f02168ec640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0210107930 0x7f0210074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:49.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.339+0000 7f02168ec640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f021010ee60 con 0x7f0210101930 2024-09-10T03:39:49.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.339+0000 7f02168ec640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f021010ece0 con 0x7f0210107930 2024-09-10T03:39:49.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.339+0000 7f020ffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0210101930 0x7f02100744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:49.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.339+0000 7f020ffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0210101930 0x7f02100744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:42140/0 (socket says 172.21.15.112:42140) 2024-09-10T03:39:49.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.339+0000 7f020ffff640 1 -- 172.21.15.112:0/782189576 learned_addr learned my addr 172.21.15.112:0/782189576 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:49.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.339+0000 7f020ffff640 1 -- 172.21.15.112:0/782189576 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0210107930 msgr2=0x7f0210074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:49.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.339+0000 7f020ffff640 1 --2- 172.21.15.112:0/782189576 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0210107930 0x7f0210074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:49.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.339+0000 7f020ffff640 1 -- 172.21.15.112:0/782189576 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0210074f70 con 0x7f0210101930 2024-09-10T03:39:49.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.340+0000 7f020ffff640 1 --2- 172.21.15.112:0/782189576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0210101930 0x7f02100744f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0200031460 tx=0x7f02000048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:49.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.340+0000 7f020d7fa640 1 -- 172.21.15.112:0/782189576 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f020003f070 con 0x7f0210101930 2024-09-10T03:39:49.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.340+0000 7f020d7fa640 1 -- 172.21.15.112:0/782189576 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0200004cf0 con 0x7f0210101930 2024-09-10T03:39:49.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.340+0000 7f02168ec640 1 -- 172.21.15.112:0/782189576 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0210075200 con 0x7f0210101930 2024-09-10T03:39:49.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.340+0000 7f020d7fa640 1 -- 172.21.15.112:0/782189576 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f0200035030 con 0x7f0210101930 2024-09-10T03:39:49.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.340+0000 7f02168ec640 1 -- 172.21.15.112:0/782189576 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02101a3e30 con 0x7f0210101930 2024-09-10T03:39:49.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.341+0000 7f02168ec640 1 -- 172.21.15.112:0/782189576 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0210103070 con 0x7f0210101930 2024-09-10T03:39:49.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.341+0000 7f020d7fa640 1 -- 172.21.15.112:0/782189576 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f0200042070 con 0x7f0210101930 2024-09-10T03:39:49.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.342+0000 7f020d7fa640 1 --2- 172.21.15.112:0/782189576 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f01f806de30 0x7f01f80702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:49.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.342+0000 7f020d7fa640 1 -- 172.21.15.112:0/782189576 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f02000ae330 con 0x7f0210101930 2024-09-10T03:39:49.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.342+0000 7f020f7fe640 1 --2- 172.21.15.112:0/782189576 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f01f806de30 0x7f01f80702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:49.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.342+0000 7f020f7fe640 1 --2- 172.21.15.112:0/782189576 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f01f806de30 0x7f01f80702f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0210075f80 tx=0x7f0204006c90 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:49.371 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.346+0000 7f020d7fa640 1 -- 172.21.15.112:0/782189576 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f020007c210 con 0x7f0210101930 2024-09-10T03:39:49.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.513+0000 7f02168ec640 1 -- 172.21.15.112:0/782189576 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0210109ea0 con 0x7f0210101930 2024-09-10T03:39:49.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.514+0000 7f020d7fa640 1 -- 172.21.15.112:0/782189576 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+11420 (secure 0 0 0) 0x7f02000ae150 con 0x7f0210101930 2024-09-10T03:39:49.515 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:39:49.515 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":39,"fsid":"513b3bc6-6f25-11ef-bcea-c7b262605968","created":"2024-09-10T03:33:31.106241+0000","modified":"2024-09-10T03:39:36.567053+0000","last_up_change":"2024-09-10T03:39:35.366928+0000","last_in_change":"2024-09-10T03:39:11.309435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T03:38:15.123477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ae9a7fcc-9d61-4914-b88f-da4fc76d5b88","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":378527782},{"type":"v1","addr":"172.21.15.112:6803","nonce":378527782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":378527782},{"type":"v1","addr":"172.21.15.112:6805","nonce":378527782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":378527782},{"type":"v1","addr":"172.21.15.112:6809","nonce":378527782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":378527782},{"type":"v1","addr":"172.21.15.112:6807","nonce":378527782}]},"public_addr":"172.21.15.112:6803/378527782","cluster_addr":"172.21.15.112:6805/378527782","heartbeat_back_addr":"172.21.15.112:6809/378527782","heartbeat_front_addr":"172.21.15.112:6807/378527782","state":["exists","up"]},{"osd":1,"uuid":"41b3eaeb-7456-42dd-9405-0770a7129bc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":305082264},{"type":"v1","addr":"172.21.15.112:6811","nonce":305082264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":305082264},{"type":"v1","addr":"172.21.15.112:6813","nonce":305082264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":305082264},{"type":"v1","addr":"172.21.15.112:6817","nonce":305082264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":305082264},{"type":"v1","addr":"172.21.15.112:6815","nonce":305082264}]},"public_addr":"172.21.15.112:6811/305082264","cluster_addr":"172.21.15.112:6813/305082264","heartbeat_back_addr":"172.21.15.112:6817/305082264","heartbeat_front_addr":"172.21.15.112:6815/305082264","state":["exists","up"]},{"osd":2,"uuid":"e7d8e980-cf86-473c-805c-61ebb62e4d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3325040968},{"type":"v1","addr":"172.21.15.112:6819","nonce":3325040968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3325040968},{"type":"v1","addr":"172.21.15.112:6821","nonce":3325040968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3325040968},{"type":"v1","addr":"172.21.15.112:6825","nonce":3325040968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3325040968},{"type":"v1","addr":"172.21.15.112:6823","nonce":3325040968}]},"public_addr":"172.21.15.112:6819/3325040968","cluster_addr":"172.21.15.112:6821/3325040968","heartbeat_back_addr":"172.21.15.112:6825/3325040968","heartbeat_front_addr":"172.21.15.112:6823/3325040968","state":["exists","up"]},{"osd":3,"uuid":"eec71a12-c779-49af-a994-414f77d0ec78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3830555794},{"type":"v1","addr":"172.21.15.169:6801","nonce":3830555794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3830555794},{"type":"v1","addr":"172.21.15.169:6803","nonce":3830555794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":3830555794},{"type":"v1","addr":"172.21.15.169:6807","nonce":3830555794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":3830555794},{"type":"v1","addr":"172.21.15.169:6805","nonce":3830555794}]},"public_addr":"172.21.15.169:6801/3830555794","cluster_addr":"172.21.15.169:6803/3830555794","heartbeat_back_addr":"172.21.15.169:6807/3830555794","heartbeat_front_addr":"172.21.15.169:6805/3830555794","state":["exists","up"]},{"osd":4,"uuid":"77c831ef-a06d-4f4f-ad2a-c2cb04613fe0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":2285032153},{"type":"v1","addr":"172.21.15.169:6809","nonce":2285032153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":2285032153},{"type":"v1","addr":"172.21.15.169:6811","nonce":2285032153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":2285032153},{"type":"v1","addr":"172.21.15.169:6815","nonce":2285032153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":2285032153},{"type":"v1","addr":"172.21.15.169:6813","nonce":2285032153}]},"public_addr":"172.21.15.169:6809/2285032153","cluster_addr":"172.21.15.169:6811/2285032153","heartbeat_back_addr":"172.21.15.169:6815/2285032153","heartbeat_front_addr":"172.21.15.169:6813/2285032153","state":["exists","up"]},{"osd":5,"uuid":"346b5969-a50d-4340-a839-ef1085e248fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":893753355},{"type":"v1","addr":"172.21.15.169:6817","nonce":893753355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":893753355},{"type":"v1","addr":"172.21.15.169:6819","nonce":893753355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":893753355},{"type":"v1","addr":"172.21.15.169:6823","nonce":893753355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":893753355},{"type":"v1","addr":"172.21.15.169:6821","nonce":893753355}]},"public_addr":"172.21.15.169:6817/893753355","cluster_addr":"172.21.15.169:6819/893753355","heartbeat_back_addr":"172.21.15.169:6823/893753355","heartbeat_front_addr":"172.21.15.169:6821/893753355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:37:09.717391+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:37:43.057817+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:38:12.048486+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:38:37.732503+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:39:06.199278+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T03:39:32.899049+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.112:6801/171256542":"2024-09-11T03:35:34.814063+0000","172.21.15.112:0/3104768277":"2024-09-11T03:34:03.009905+0000","172.21.15.112:6800/171256542":"2024-09-11T03:35:34.814063+0000","172.21.15.112:6800/3423408536":"2024-09-11T03:34:03.009905+0000","172.21.15.112:0/3329867731":"2024-09-11T03:34:03.009905+0000","172.21.15.112:0/3444044994":"2024-09-11T03:34:03.009905+0000","172.21.15.112:0/288016527":"2024-09-11T03:34:33.723096+0000","172.21.15.112:0/3338248392":"2024-09-11T03:34:33.723096+0000","172.21.15.112:6801/4183006970":"2024-09-11T03:34:33.723096+0000","172.21.15.112:0/4078185439":"2024-09-11T03:34:33.723096+0000","172.21.15.112:6800/4183006970":"2024-09-11T03:34:33.723096+0000","172.21.15.112:0/3410252058":"2024-09-11T03:35:34.814063+0000","172.21.15.112:6801/3423408536":"2024-09-11T03:34:03.009905+0000","172.21.15.112:0/1394011979":"2024-09-11T03:35:34.814063+0000","172.21.15.112:0/522295971":"2024-09-11T03:35:34.814063+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T03:39:49.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.516+0000 7f02168ec640 1 -- 172.21.15.112:0/782189576 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f01f806de30 msgr2=0x7f01f80702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:49.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.516+0000 7f02168ec640 1 --2- 172.21.15.112:0/782189576 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f01f806de30 0x7f01f80702f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0210075f80 tx=0x7f0204006c90 comp rx=0 tx=0).stop 2024-09-10T03:39:49.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.516+0000 7f02168ec640 1 -- 172.21.15.112:0/782189576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0210101930 msgr2=0x7f02100744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:49.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.516+0000 7f02168ec640 1 --2- 172.21.15.112:0/782189576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0210101930 0x7f02100744f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0200031460 tx=0x7f02000048f0 comp rx=0 tx=0).stop 2024-09-10T03:39:49.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.516+0000 7f02168ec640 1 -- 172.21.15.112:0/782189576 shutdown_connections 2024-09-10T03:39:49.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.516+0000 7f02168ec640 1 --2- 172.21.15.112:0/782189576 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f01f806de30 0x7f01f80702f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:49.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.516+0000 7f02168ec640 1 --2- 172.21.15.112:0/782189576 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0210107930 0x7f0210074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:49.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.516+0000 7f02168ec640 1 --2- 172.21.15.112:0/782189576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0210101930 0x7f02100744f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:49.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.516+0000 7f02168ec640 1 -- 172.21.15.112:0/782189576 >> 172.21.15.112:0/782189576 conn(0x7f02100fd630 msgr2=0x7f02100fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:49.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.516+0000 7f02168ec640 1 -- 172.21.15.112:0/782189576 shutdown_connections 2024-09-10T03:39:49.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:49.516+0000 7f02168ec640 1 -- 172.21.15.112:0/782189576 wait complete. 2024-09-10T03:39:50.141 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-10T03:39:50.142 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-10T03:39:50.142 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-10T03:39:50.143 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-10T03:39:50.143 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-10T03:39:50.143 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-10T03:39:50.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:49 smithi169 ceph-mon[35668]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:50.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:49 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/782189576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T03:39:50.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:49 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:39:50.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:49 smithi112 ceph-mon[27737]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:50.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:49 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/782189576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T03:39:50.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:49 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:39:50.452 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:50.466 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:51.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.106+0000 7f3b1f6f7640 1 -- 172.21.15.112:0/1560842307 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b181008d0 msgr2=0x7f3b18100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:51.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.106+0000 7f3b1f6f7640 1 --2- 172.21.15.112:0/1560842307 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b181008d0 0x7f3b18100cb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3b14009880 tx=0x7f3b1402a530 comp rx=0 tx=0).stop 2024-09-10T03:39:51.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.106+0000 7f3b1f6f7640 1 -- 172.21.15.112:0/1560842307 shutdown_connections 2024-09-10T03:39:51.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.106+0000 7f3b1f6f7640 1 --2- 172.21.15.112:0/1560842307 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b181011f0 0x7f3b181056c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:51.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.106+0000 7f3b1f6f7640 1 --2- 172.21.15.112:0/1560842307 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b181008d0 0x7f3b18100cb0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:51.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.106+0000 7f3b1f6f7640 1 -- 172.21.15.112:0/1560842307 >> 172.21.15.112:0/1560842307 conn(0x7f3b180fc520 msgr2=0x7f3b180fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:51.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.106+0000 7f3b1f6f7640 1 -- 172.21.15.112:0/1560842307 shutdown_connections 2024-09-10T03:39:51.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.106+0000 7f3b1f6f7640 1 -- 172.21.15.112:0/1560842307 wait complete. 2024-09-10T03:39:51.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.107+0000 7f3b1f6f7640 1 Processor -- start 2024-09-10T03:39:51.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.107+0000 7f3b1f6f7640 1 -- start start 2024-09-10T03:39:51.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.107+0000 7f3b1f6f7640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b181008d0 0x7f3b1819dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:51.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.107+0000 7f3b1f6f7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b181011f0 0x7f3b1819e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:51.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.107+0000 7f3b1f6f7640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3b18115470 con 0x7f3b181008d0 2024-09-10T03:39:51.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.107+0000 7f3b1f6f7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f3b181152f0 con 0x7f3b181011f0 2024-09-10T03:39:51.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.107+0000 7f3b1d493640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b181008d0 0x7f3b1819dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:51.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.107+0000 7f3b1d493640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b181008d0 0x7f3b1819dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:42156/0 (socket says 172.21.15.112:42156) 2024-09-10T03:39:51.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.107+0000 7f3b1d493640 1 -- 172.21.15.112:0/4219194256 learned_addr learned my addr 172.21.15.112:0/4219194256 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:51.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.107+0000 7f3b1cc92640 1 --2- 172.21.15.112:0/4219194256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b181011f0 0x7f3b1819e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:51.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.107+0000 7f3b1d493640 1 -- 172.21.15.112:0/4219194256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b181011f0 msgr2=0x7f3b1819e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:51.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.107+0000 7f3b1d493640 1 --2- 172.21.15.112:0/4219194256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b181011f0 0x7f3b1819e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:51.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.107+0000 7f3b1d493640 1 -- 172.21.15.112:0/4219194256 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b181a28c0 con 0x7f3b181008d0 2024-09-10T03:39:51.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.107+0000 7f3b1cc92640 1 --2- 172.21.15.112:0/4219194256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b181011f0 0x7f3b1819e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T03:39:51.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.107+0000 7f3b1d493640 1 --2- 172.21.15.112:0/4219194256 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b181008d0 0x7f3b1819dff0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3b14031460 tx=0x7f3b140048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:51.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.108+0000 7f3b0e7fc640 1 -- 172.21.15.112:0/4219194256 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3b1403f070 con 0x7f3b181008d0 2024-09-10T03:39:51.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.108+0000 7f3b0e7fc640 1 -- 172.21.15.112:0/4219194256 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3b140044d0 con 0x7f3b181008d0 2024-09-10T03:39:51.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.108+0000 7f3b1f6f7640 1 -- 172.21.15.112:0/4219194256 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b181a2b50 con 0x7f3b181008d0 2024-09-10T03:39:51.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.108+0000 7f3b0e7fc640 1 -- 172.21.15.112:0/4219194256 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3b14035030 con 0x7f3b181008d0 2024-09-10T03:39:51.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.108+0000 7f3b1f6f7640 1 -- 172.21.15.112:0/4219194256 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b181a3080 con 0x7f3b181008d0 2024-09-10T03:39:51.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.109+0000 7f3b1f6f7640 1 -- 172.21.15.112:0/4219194256 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3b181a3640 con 0x7f3b181008d0 2024-09-10T03:39:51.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.109+0000 7f3b0e7fc640 1 -- 172.21.15.112:0/4219194256 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f3b14042070 con 0x7f3b181008d0 2024-09-10T03:39:51.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.110+0000 7f3b0e7fc640 1 --2- 172.21.15.112:0/4219194256 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3b0406de30 0x7f3b040702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:51.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.110+0000 7f3b1cc92640 1 --2- 172.21.15.112:0/4219194256 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3b0406de30 0x7f3b040702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:51.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.110+0000 7f3b0e7fc640 1 -- 172.21.15.112:0/4219194256 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f3b140aea20 con 0x7f3b181008d0 2024-09-10T03:39:51.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.110+0000 7f3b0e7fc640 1 --2- 172.21.15.112:0/4219194256 >> [v2:172.21.15.169:6808/2285032153,v1:172.21.15.169:6809/2285032153] conn(0x7f3b04073c20 0x7f3b04076040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:51.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.110+0000 7f3b0e7fc640 1 -- 172.21.15.112:0/4219194256 --> [v2:172.21.15.169:6808/2285032153,v1:172.21.15.169:6809/2285032153] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3b14003e30 con 0x7f3b04073c20 2024-09-10T03:39:51.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.110+0000 7f3b0e7fc640 1 -- 172.21.15.112:0/4219194256 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f3b140aeec0 con 0x7f3b181008d0 2024-09-10T03:39:51.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.110+0000 7f3b1dc94640 1 --2- 172.21.15.112:0/4219194256 >> [v2:172.21.15.169:6808/2285032153,v1:172.21.15.169:6809/2285032153] conn(0x7f3b04073c20 0x7f3b04076040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:51.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.110+0000 7f3b1cc92640 1 --2- 172.21.15.112:0/4219194256 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3b0406de30 0x7f3b040702f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3b08006fd0 tx=0x7f3b08008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:51.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.111+0000 7f3b1dc94640 1 --2- 172.21.15.112:0/4219194256 >> [v2:172.21.15.169:6808/2285032153,v1:172.21.15.169:6809/2285032153] conn(0x7f3b04073c20 0x7f3b04076040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:51.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.111+0000 7f3b0e7fc640 1 -- 172.21.15.112:0/4219194256 <== osd.4 v2:172.21.15.169:6808/2285032153 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3b14003e30 con 0x7f3b04073c20 2024-09-10T03:39:51.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.125+0000 7f3b1f6f7640 1 -- 172.21.15.112:0/4219194256 --> [v2:172.21.15.169:6808/2285032153,v1:172.21.15.169:6809/2285032153] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3b18062880 con 0x7f3b04073c20 2024-09-10T03:39:51.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.125+0000 7f3b0e7fc640 1 -- 172.21.15.112:0/4219194256 <== osd.4 v2:172.21.15.169:6808/2285032153 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3b18062880 con 0x7f3b04073c20 2024-09-10T03:39:51.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.125+0000 7f3b1f6f7640 1 -- 172.21.15.112:0/4219194256 >> [v2:172.21.15.169:6808/2285032153,v1:172.21.15.169:6809/2285032153] conn(0x7f3b04073c20 msgr2=0x7f3b04076040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:51.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.125+0000 7f3b1f6f7640 1 --2- 172.21.15.112:0/4219194256 >> [v2:172.21.15.169:6808/2285032153,v1:172.21.15.169:6809/2285032153] conn(0x7f3b04073c20 0x7f3b04076040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:51.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.125+0000 7f3b1f6f7640 1 -- 172.21.15.112:0/4219194256 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3b0406de30 msgr2=0x7f3b040702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:51.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.125+0000 7f3b1f6f7640 1 --2- 172.21.15.112:0/4219194256 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3b0406de30 0x7f3b040702f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3b08006fd0 tx=0x7f3b08008040 comp rx=0 tx=0).stop 2024-09-10T03:39:51.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.125+0000 7f3b1f6f7640 1 -- 172.21.15.112:0/4219194256 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b181008d0 msgr2=0x7f3b1819dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:51.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.125+0000 7f3b1f6f7640 1 --2- 172.21.15.112:0/4219194256 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b181008d0 0x7f3b1819dff0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3b14031460 tx=0x7f3b140048f0 comp rx=0 tx=0).stop 2024-09-10T03:39:51.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.125+0000 7f3b1f6f7640 1 -- 172.21.15.112:0/4219194256 shutdown_connections 2024-09-10T03:39:51.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.125+0000 7f3b1f6f7640 1 --2- 172.21.15.112:0/4219194256 >> [v2:172.21.15.169:6808/2285032153,v1:172.21.15.169:6809/2285032153] conn(0x7f3b04073c20 0x7f3b04076040 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:51.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.125+0000 7f3b1f6f7640 1 --2- 172.21.15.112:0/4219194256 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3b0406de30 0x7f3b040702f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:51.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.125+0000 7f3b1f6f7640 1 --2- 172.21.15.112:0/4219194256 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3b181011f0 0x7f3b1819e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:51.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.125+0000 7f3b1f6f7640 1 --2- 172.21.15.112:0/4219194256 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b181008d0 0x7f3b1819dff0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:51.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.125+0000 7f3b1f6f7640 1 -- 172.21.15.112:0/4219194256 >> 172.21.15.112:0/4219194256 conn(0x7f3b180fc520 msgr2=0x7f3b18112b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:51.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.126+0000 7f3b1f6f7640 1 -- 172.21.15.112:0/4219194256 shutdown_connections 2024-09-10T03:39:51.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.126+0000 7f3b1f6f7640 1 -- 172.21.15.112:0/4219194256 wait complete. 2024-09-10T03:39:51.348 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:51.348 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:51.349 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:51.349 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:51.991 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:51 smithi112 ceph-mon[27737]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:52.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.933+0000 7fb301a00640 1 -- 172.21.15.112:0/1722475684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2fc100b80 msgr2=0x7fb2fc104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:52.092 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.933+0000 7fb301a00640 1 --2- 172.21.15.112:0/1722475684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2fc100b80 0x7fb2fc104bd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0009880 tx=0x7fb2f002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:39:52.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.934+0000 7fb301a00640 1 -- 172.21.15.112:0/1722475684 shutdown_connections 2024-09-10T03:39:52.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.934+0000 7fb301a00640 1 --2- 172.21.15.112:0/1722475684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2fc100b80 0x7fb2fc104bd0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:52.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.934+0000 7fb301a00640 1 --2- 172.21.15.112:0/1722475684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2fc1001d0 0x7fb2fc1005b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:52.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.934+0000 7fb301a00640 1 -- 172.21.15.112:0/1722475684 >> 172.21.15.112:0/1722475684 conn(0x7fb2fc0fbc50 msgr2=0x7fb2fc0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:52.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.934+0000 7fb301a00640 1 -- 172.21.15.112:0/1722475684 shutdown_connections 2024-09-10T03:39:52.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.934+0000 7fb301a00640 1 -- 172.21.15.112:0/1722475684 wait complete. 2024-09-10T03:39:52.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.934+0000 7fb301a00640 1 Processor -- start 2024-09-10T03:39:52.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.935+0000 7fb301a00640 1 -- start start 2024-09-10T03:39:52.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.935+0000 7fb301a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2fc1001d0 0x7fb2fc19d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:52.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.935+0000 7fb301a00640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2fc100b80 0x7fb2fc19dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:52.093 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.935+0000 7fb301a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb2fc10d450 con 0x7fb2fc1001d0 2024-09-10T03:39:52.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.935+0000 7fb301a00640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb2fc10d2d0 con 0x7fb2fc100b80 2024-09-10T03:39:52.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.935+0000 7fb2faffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2fc1001d0 0x7fb2fc19d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:52.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.935+0000 7fb2faffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2fc1001d0 0x7fb2fc19d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:42174/0 (socket says 172.21.15.112:42174) 2024-09-10T03:39:52.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.935+0000 7fb2faffd640 1 -- 172.21.15.112:0/163240669 learned_addr learned my addr 172.21.15.112:0/163240669 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:52.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.935+0000 7fb2fa7fc640 1 --2- 172.21.15.112:0/163240669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2fc100b80 0x7fb2fc19dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:52.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.935+0000 7fb2faffd640 1 -- 172.21.15.112:0/163240669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2fc100b80 msgr2=0x7fb2fc19dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:52.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.935+0000 7fb2faffd640 1 --2- 172.21.15.112:0/163240669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2fc100b80 0x7fb2fc19dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:52.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.935+0000 7fb2faffd640 1 -- 172.21.15.112:0/163240669 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2fc197860 con 0x7fb2fc1001d0 2024-09-10T03:39:52.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.935+0000 7fb2fa7fc640 1 --2- 172.21.15.112:0/163240669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2fc100b80 0x7fb2fc19dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T03:39:52.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.935+0000 7fb2faffd640 1 --2- 172.21.15.112:0/163240669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2fc1001d0 0x7fb2fc19d6a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec00e970 tx=0x7fb2ec00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:52.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.935+0000 7fb3009fe640 1 -- 172.21.15.112:0/163240669 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb2ec00cac0 con 0x7fb2fc1001d0 2024-09-10T03:39:52.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.935+0000 7fb301a00640 1 -- 172.21.15.112:0/163240669 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2fc197b50 con 0x7fb2fc1001d0 2024-09-10T03:39:52.094 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.935+0000 7fb3009fe640 1 -- 172.21.15.112:0/163240669 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb2ec0054e0 con 0x7fb2fc1001d0 2024-09-10T03:39:52.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.936+0000 7fb3009fe640 1 -- 172.21.15.112:0/163240669 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb2ec010430 con 0x7fb2fc1001d0 2024-09-10T03:39:52.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.936+0000 7fb301a00640 1 -- 172.21.15.112:0/163240669 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2fc1980b0 con 0x7fb2fc1001d0 2024-09-10T03:39:52.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.936+0000 7fb301a00640 1 -- 172.21.15.112:0/163240669 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb2fc198650 con 0x7fb2fc1001d0 2024-09-10T03:39:52.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.937+0000 7fb3009fe640 1 -- 172.21.15.112:0/163240669 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fb2ec004ec0 con 0x7fb2fc1001d0 2024-09-10T03:39:52.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.937+0000 7fb3009fe640 1 --2- 172.21.15.112:0/163240669 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb2e406de30 0x7fb2e40702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:52.095 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.937+0000 7fb3009fe640 1 -- 172.21.15.112:0/163240669 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7fb2ec08df40 con 0x7fb2fc1001d0 2024-09-10T03:39:52.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.937+0000 7fb2fa7fc640 1 --2- 172.21.15.112:0/163240669 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb2e406de30 0x7fb2e40702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:52.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.938+0000 7fb3009fe640 1 --2- 172.21.15.112:0/163240669 >> [v2:172.21.15.169:6800/3830555794,v1:172.21.15.169:6801/3830555794] conn(0x7fb2e4073c20 0x7fb2e4076040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:52.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.938+0000 7fb3009fe640 1 -- 172.21.15.112:0/163240669 --> [v2:172.21.15.169:6800/3830555794,v1:172.21.15.169:6801/3830555794] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb2e40766f0 con 0x7fb2e4073c20 2024-09-10T03:39:52.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.938+0000 7fb3009fe640 1 -- 172.21.15.112:0/163240669 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fb2ec095050 con 0x7fb2fc1001d0 2024-09-10T03:39:52.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.938+0000 7fb2fa7fc640 1 --2- 172.21.15.112:0/163240669 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb2e406de30 0x7fb2e40702f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc199050 tx=0x7fb2f0031400 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:52.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.938+0000 7fb2fb7fe640 1 --2- 172.21.15.112:0/163240669 >> [v2:172.21.15.169:6800/3830555794,v1:172.21.15.169:6801/3830555794] conn(0x7fb2e4073c20 0x7fb2e4076040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:52.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.938+0000 7fb2fb7fe640 1 --2- 172.21.15.112:0/163240669 >> [v2:172.21.15.169:6800/3830555794,v1:172.21.15.169:6801/3830555794] conn(0x7fb2e4073c20 0x7fb2e4076040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:52.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.939+0000 7fb3009fe640 1 -- 172.21.15.112:0/163240669 <== osd.3 v2:172.21.15.169:6800/3830555794 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb2e40766f0 con 0x7fb2e4073c20 2024-09-10T03:39:52.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.953+0000 7fb301a00640 1 -- 172.21.15.112:0/163240669 --> [v2:172.21.15.169:6800/3830555794,v1:172.21.15.169:6801/3830555794] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb2fc062880 con 0x7fb2e4073c20 2024-09-10T03:39:52.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.953+0000 7fb3009fe640 1 -- 172.21.15.112:0/163240669 <== osd.3 v2:172.21.15.169:6800/3830555794 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb2fc062880 con 0x7fb2e4073c20 2024-09-10T03:39:52.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.953+0000 7fb301a00640 1 -- 172.21.15.112:0/163240669 >> [v2:172.21.15.169:6800/3830555794,v1:172.21.15.169:6801/3830555794] conn(0x7fb2e4073c20 msgr2=0x7fb2e4076040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:52.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.953+0000 7fb301a00640 1 --2- 172.21.15.112:0/163240669 >> [v2:172.21.15.169:6800/3830555794,v1:172.21.15.169:6801/3830555794] conn(0x7fb2e4073c20 0x7fb2e4076040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:52.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.954+0000 7fb301a00640 1 -- 172.21.15.112:0/163240669 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb2e406de30 msgr2=0x7fb2e40702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:52.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.954+0000 7fb301a00640 1 --2- 172.21.15.112:0/163240669 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb2e406de30 0x7fb2e40702f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc199050 tx=0x7fb2f0031400 comp rx=0 tx=0).stop 2024-09-10T03:39:52.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.954+0000 7fb301a00640 1 -- 172.21.15.112:0/163240669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2fc1001d0 msgr2=0x7fb2fc19d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:52.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.954+0000 7fb301a00640 1 --2- 172.21.15.112:0/163240669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2fc1001d0 0x7fb2fc19d6a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec00e970 tx=0x7fb2ec00ed70 comp rx=0 tx=0).stop 2024-09-10T03:39:52.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.954+0000 7fb301a00640 1 -- 172.21.15.112:0/163240669 shutdown_connections 2024-09-10T03:39:52.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.954+0000 7fb301a00640 1 --2- 172.21.15.112:0/163240669 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb2e406de30 0x7fb2e40702f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:52.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.954+0000 7fb301a00640 1 --2- 172.21.15.112:0/163240669 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2fc100b80 0x7fb2fc19dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:52.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.954+0000 7fb301a00640 1 --2- 172.21.15.112:0/163240669 >> [v2:172.21.15.169:6800/3830555794,v1:172.21.15.169:6801/3830555794] conn(0x7fb2e4073c20 0x7fb2e4076040 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:52.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.954+0000 7fb301a00640 1 --2- 172.21.15.112:0/163240669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2fc1001d0 0x7fb2fc19d6a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:52.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.954+0000 7fb301a00640 1 -- 172.21.15.112:0/163240669 >> 172.21.15.112:0/163240669 conn(0x7fb2fc0fbc50 msgr2=0x7fb2fc103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:52.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.954+0000 7fb301a00640 1 -- 172.21.15.112:0/163240669 shutdown_connections 2024-09-10T03:39:52.097 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:51.954+0000 7fb301a00640 1 -- 172.21.15.112:0/163240669 wait complete. 2024-09-10T03:39:52.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:51 smithi169 ceph-mon[35668]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:52.726 INFO:teuthology.orchestra.run.smithi112.stdout:107374182417 2024-09-10T03:39:52.726 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-10T03:39:52.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.855+0000 7f14fd02f640 1 -- 172.21.15.112:0/2058080280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14f8102ae0 msgr2=0x7f14f8102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:52.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.855+0000 7f14fd02f640 1 --2- 172.21.15.112:0/2058080280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14f8102ae0 0x7f14f8102ec0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009880 tx=0x7f14e802a530 comp rx=0 tx=0).stop 2024-09-10T03:39:52.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.855+0000 7f14fd02f640 1 -- 172.21.15.112:0/2058080280 shutdown_connections 2024-09-10T03:39:52.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.855+0000 7f14fd02f640 1 --2- 172.21.15.112:0/2058080280 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14f8103400 0x7f14f810b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:52.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.855+0000 7f14fd02f640 1 --2- 172.21.15.112:0/2058080280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14f8102ae0 0x7f14f8102ec0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:52.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.855+0000 7f14fd02f640 1 -- 172.21.15.112:0/2058080280 >> 172.21.15.112:0/2058080280 conn(0x7f14f80fc520 msgr2=0x7f14f80fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:52.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.855+0000 7f14fd02f640 1 -- 172.21.15.112:0/2058080280 shutdown_connections 2024-09-10T03:39:52.856 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.855+0000 7f14fd02f640 1 -- 172.21.15.112:0/2058080280 wait complete. 2024-09-10T03:39:52.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.855+0000 7f14fd02f640 1 Processor -- start 2024-09-10T03:39:52.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.856+0000 7f14fd02f640 1 -- start start 2024-09-10T03:39:52.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.856+0000 7f14fd02f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14f8102ae0 0x7f14f8100740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:52.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.856+0000 7f14fd02f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14f8103400 0x7f14f8100c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:52.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.856+0000 7f14fd02f640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f14f810ee70 con 0x7f14f8103400 2024-09-10T03:39:52.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.856+0000 7f14fd02f640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f14f810ecf0 con 0x7f14f8102ae0 2024-09-10T03:39:52.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.856+0000 7f14f659c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14f8103400 0x7f14f8100c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:52.858 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.856+0000 7f14f659c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14f8103400 0x7f14f8100c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:42198/0 (socket says 172.21.15.112:42198) 2024-09-10T03:39:52.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.856+0000 7f14f659c640 1 -- 172.21.15.112:0/3829837259 learned_addr learned my addr 172.21.15.112:0/3829837259 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:52.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.856+0000 7f14f6d9d640 1 --2- 172.21.15.112:0/3829837259 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14f8102ae0 0x7f14f8100740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:52.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.856+0000 7f14f659c640 1 -- 172.21.15.112:0/3829837259 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14f8102ae0 msgr2=0x7f14f8100740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:52.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.856+0000 7f14f659c640 1 --2- 172.21.15.112:0/3829837259 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14f8102ae0 0x7f14f8100740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:52.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.856+0000 7f14f659c640 1 -- 172.21.15.112:0/3829837259 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f81011c0 con 0x7f14f8103400 2024-09-10T03:39:52.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.857+0000 7f14f659c640 1 --2- 172.21.15.112:0/3829837259 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14f8103400 0x7f14f8100c80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00b370 tx=0x7f14ec00b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:52.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.857+0000 7f14dffff640 1 -- 172.21.15.112:0/3829837259 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f14ec015020 con 0x7f14f8103400 2024-09-10T03:39:52.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.857+0000 7f14fd02f640 1 -- 172.21.15.112:0/3829837259 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14f81014b0 con 0x7f14f8103400 2024-09-10T03:39:52.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.857+0000 7f14dffff640 1 -- 172.21.15.112:0/3829837259 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f14ec011af0 con 0x7f14f8103400 2024-09-10T03:39:52.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.857+0000 7f14dffff640 1 -- 172.21.15.112:0/3829837259 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f14ec010d60 con 0x7f14f8103400 2024-09-10T03:39:52.859 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.857+0000 7f14fd02f640 1 -- 172.21.15.112:0/3829837259 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14f81a3ee0 con 0x7f14f8103400 2024-09-10T03:39:52.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.858+0000 7f14fd02f640 1 -- 172.21.15.112:0/3829837259 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f14f80628c0 con 0x7f14f8103400 2024-09-10T03:39:52.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.858+0000 7f14dffff640 1 -- 172.21.15.112:0/3829837259 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f14ec018020 con 0x7f14f8103400 2024-09-10T03:39:52.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.859+0000 7f14dffff640 1 --2- 172.21.15.112:0/3829837259 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f14e006de30 0x7f14e00702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:52.860 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.859+0000 7f14dffff640 1 -- 172.21.15.112:0/3829837259 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f14ec08d270 con 0x7f14f8103400 2024-09-10T03:39:52.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.859+0000 7f14f6d9d640 1 --2- 172.21.15.112:0/3829837259 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f14e006de30 0x7f14e00702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:52.861 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.859+0000 7f14dffff640 1 --2- 172.21.15.112:0/3829837259 >> [v2:172.21.15.169:6816/893753355,v1:172.21.15.169:6817/893753355] conn(0x7f14e0073c20 0x7f14e0076040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:52.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.859+0000 7f14dffff640 1 -- 172.21.15.112:0/3829837259 --> [v2:172.21.15.169:6816/893753355,v1:172.21.15.169:6817/893753355] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f14ec014e70 con 0x7f14e0073c20 2024-09-10T03:39:52.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.859+0000 7f14dffff640 1 -- 172.21.15.112:0/3829837259 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f14ec094050 con 0x7f14f8103400 2024-09-10T03:39:52.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.859+0000 7f14f6d9d640 1 --2- 172.21.15.112:0/3829837259 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f14e006de30 0x7f14e00702f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f14e80096c0 tx=0x7f14e8031400 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:52.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.859+0000 7f14f759e640 1 --2- 172.21.15.112:0/3829837259 >> [v2:172.21.15.169:6816/893753355,v1:172.21.15.169:6817/893753355] conn(0x7f14e0073c20 0x7f14e0076040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:52.862 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.860+0000 7f14f759e640 1 --2- 172.21.15.112:0/3829837259 >> [v2:172.21.15.169:6816/893753355,v1:172.21.15.169:6817/893753355] conn(0x7f14e0073c20 0x7f14e0076040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:52.863 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.861+0000 7f14dffff640 1 -- 172.21.15.112:0/3829837259 <== osd.5 v2:172.21.15.169:6816/893753355 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14ec014e70 con 0x7f14e0073c20 2024-09-10T03:39:52.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.874+0000 7f14fd02f640 1 -- 172.21.15.112:0/3829837259 --> [v2:172.21.15.169:6816/893753355,v1:172.21.15.169:6817/893753355] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f14f8062d10 con 0x7f14e0073c20 2024-09-10T03:39:52.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.874+0000 7f14dffff640 1 -- 172.21.15.112:0/3829837259 <== osd.5 v2:172.21.15.169:6816/893753355 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f14f8062d10 con 0x7f14e0073c20 2024-09-10T03:39:52.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.875+0000 7f14fd02f640 1 -- 172.21.15.112:0/3829837259 >> [v2:172.21.15.169:6816/893753355,v1:172.21.15.169:6817/893753355] conn(0x7f14e0073c20 msgr2=0x7f14e0076040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:52.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.875+0000 7f14fd02f640 1 --2- 172.21.15.112:0/3829837259 >> [v2:172.21.15.169:6816/893753355,v1:172.21.15.169:6817/893753355] conn(0x7f14e0073c20 0x7f14e0076040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:52.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.875+0000 7f14fd02f640 1 -- 172.21.15.112:0/3829837259 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f14e006de30 msgr2=0x7f14e00702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:52.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.875+0000 7f14fd02f640 1 --2- 172.21.15.112:0/3829837259 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f14e006de30 0x7f14e00702f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f14e80096c0 tx=0x7f14e8031400 comp rx=0 tx=0).stop 2024-09-10T03:39:52.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.875+0000 7f14fd02f640 1 -- 172.21.15.112:0/3829837259 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14f8103400 msgr2=0x7f14f8100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:52.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.875+0000 7f14fd02f640 1 --2- 172.21.15.112:0/3829837259 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14f8103400 0x7f14f8100c80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f14ec00b370 tx=0x7f14ec00b830 comp rx=0 tx=0).stop 2024-09-10T03:39:52.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.875+0000 7f14fd02f640 1 -- 172.21.15.112:0/3829837259 shutdown_connections 2024-09-10T03:39:52.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.875+0000 7f14fd02f640 1 --2- 172.21.15.112:0/3829837259 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f14e006de30 0x7f14e00702f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:52.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.875+0000 7f14fd02f640 1 --2- 172.21.15.112:0/3829837259 >> [v2:172.21.15.169:6816/893753355,v1:172.21.15.169:6817/893753355] conn(0x7f14e0073c20 0x7f14e0076040 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:52.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.875+0000 7f14fd02f640 1 --2- 172.21.15.112:0/3829837259 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f14f8103400 0x7f14f8100c80 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:52.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.875+0000 7f14fd02f640 1 --2- 172.21.15.112:0/3829837259 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14f8102ae0 0x7f14f8100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:52.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.875+0000 7f14fd02f640 1 -- 172.21.15.112:0/3829837259 >> 172.21.15.112:0/3829837259 conn(0x7f14f80fc520 msgr2=0x7f14f8108eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:52.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.875+0000 7f14fd02f640 1 -- 172.21.15.112:0/3829837259 shutdown_connections 2024-09-10T03:39:52.877 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:52.875+0000 7f14fd02f640 1 -- 172.21.15.112:0/3829837259 wait complete. 2024-09-10T03:39:54.053 INFO:teuthology.orchestra.run.smithi112.stdout:141733920779 2024-09-10T03:39:54.054 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-10T03:39:54.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.290+0000 7f5165432640 1 -- 172.21.15.112:0/1115929352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51601067e0 msgr2=0x7f5160106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:54.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.290+0000 7f5165432640 1 --2- 172.21.15.112:0/1115929352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51601067e0 0x7f5160106bc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5150009c20 tx=0x7f515002a970 comp rx=0 tx=0).stop 2024-09-10T03:39:54.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.290+0000 7f5165432640 1 -- 172.21.15.112:0/1115929352 shutdown_connections 2024-09-10T03:39:54.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.290+0000 7f5165432640 1 --2- 172.21.15.112:0/1115929352 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51601007e0 0x7f5160100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.290+0000 7f5165432640 1 --2- 172.21.15.112:0/1115929352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51601067e0 0x7f5160106bc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.290+0000 7f5165432640 1 -- 172.21.15.112:0/1115929352 >> 172.21.15.112:0/1115929352 conn(0x7f51600fc4c0 msgr2=0x7f51600fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:54.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.290+0000 7f5165432640 1 -- 172.21.15.112:0/1115929352 shutdown_connections 2024-09-10T03:39:54.135 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.290+0000 7f5165432640 1 -- 172.21.15.112:0/1115929352 wait complete. 2024-09-10T03:39:54.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.291+0000 7f5165432640 1 Processor -- start 2024-09-10T03:39:54.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.291+0000 7f5165432640 1 -- start start 2024-09-10T03:39:54.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.291+0000 7f5165432640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51601007e0 0x7f5160198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:54.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.291+0000 7f5165432640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51601067e0 0x7f5160198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:54.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.291+0000 7f5165432640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f516010dd10 con 0x7f51601067e0 2024-09-10T03:39:54.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.291+0000 7f5165432640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f516010db90 con 0x7f51601007e0 2024-09-10T03:39:54.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.291+0000 7f515e7fc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51601067e0 0x7f5160198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:54.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.292+0000 7f515e7fc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51601067e0 0x7f5160198e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:42206/0 (socket says 172.21.15.112:42206) 2024-09-10T03:39:54.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.292+0000 7f515e7fc640 1 -- 172.21.15.112:0/4280430905 learned_addr learned my addr 172.21.15.112:0/4280430905 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:54.136 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.292+0000 7f515effd640 1 --2- 172.21.15.112:0/4280430905 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51601007e0 0x7f5160198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.292+0000 7f515e7fc640 1 -- 172.21.15.112:0/4280430905 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51601007e0 msgr2=0x7f5160198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.292+0000 7f515e7fc640 1 --2- 172.21.15.112:0/4280430905 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51601007e0 0x7f5160198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.292+0000 7f515e7fc640 1 -- 172.21.15.112:0/4280430905 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51601994b0 con 0x7f51601067e0 2024-09-10T03:39:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.292+0000 7f515e7fc640 1 --2- 172.21.15.112:0/4280430905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51601067e0 0x7f5160198e60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f514800b420 tx=0x7f514800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.292+0000 7f5157fff640 1 -- 172.21.15.112:0/4280430905 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f5148011630 con 0x7f51601067e0 2024-09-10T03:39:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.292+0000 7f5157fff640 1 -- 172.21.15.112:0/4280430905 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5148016070 con 0x7f51601067e0 2024-09-10T03:39:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.292+0000 7f5165432640 1 -- 172.21.15.112:0/4280430905 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51601a1990 con 0x7f51601067e0 2024-09-10T03:39:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.292+0000 7f5157fff640 1 -- 172.21.15.112:0/4280430905 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f5148010d40 con 0x7f51601067e0 2024-09-10T03:39:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.292+0000 7f5165432640 1 -- 172.21.15.112:0/4280430905 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51601a1d90 con 0x7f51601067e0 2024-09-10T03:39:54.137 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.293+0000 7f5165432640 1 -- 172.21.15.112:0/4280430905 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f51600628c0 con 0x7f51601067e0 2024-09-10T03:39:54.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.293+0000 7f5157fff640 1 -- 172.21.15.112:0/4280430905 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f514801b460 con 0x7f51601067e0 2024-09-10T03:39:54.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.294+0000 7f5157fff640 1 --2- 172.21.15.112:0/4280430905 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f514c06de80 0x7f514c070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:54.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.294+0000 7f515effd640 1 --2- 172.21.15.112:0/4280430905 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f514c06de80 0x7f514c070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:54.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.294+0000 7f5157fff640 1 -- 172.21.15.112:0/4280430905 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f514808d010 con 0x7f51601067e0 2024-09-10T03:39:54.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.294+0000 7f5157fff640 1 --2- 172.21.15.112:0/4280430905 >> [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782] conn(0x7f514c073c70 0x7f514c076090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:54.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.294+0000 7f5157fff640 1 -- 172.21.15.112:0/4280430905 --> [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f514c076740 con 0x7f514c073c70 2024-09-10T03:39:54.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.294+0000 7f5157fff640 1 -- 172.21.15.112:0/4280430905 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f5148094050 con 0x7f51601067e0 2024-09-10T03:39:54.138 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.294+0000 7f515f7fe640 1 --2- 172.21.15.112:0/4280430905 >> [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782] conn(0x7f514c073c70 0x7f514c076090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:54.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.294+0000 7f515effd640 1 --2- 172.21.15.112:0/4280430905 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f514c06de80 0x7f514c070340 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5150009a60 tx=0x7f515002f9d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:54.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.295+0000 7f515f7fe640 1 --2- 172.21.15.112:0/4280430905 >> [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782] conn(0x7f514c073c70 0x7f514c076090 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:54.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.295+0000 7f5157fff640 1 -- 172.21.15.112:0/4280430905 <== osd.0 v2:172.21.15.112:6802/378527782 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f514c076740 con 0x7f514c073c70 2024-09-10T03:39:54.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.308+0000 7f5165432640 1 -- 172.21.15.112:0/4280430905 --> [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5160062cc0 con 0x7f514c073c70 2024-09-10T03:39:54.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.309+0000 7f5157fff640 1 -- 172.21.15.112:0/4280430905 <== osd.0 v2:172.21.15.112:6802/378527782 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5160062cc0 con 0x7f514c073c70 2024-09-10T03:39:54.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.309+0000 7f5165432640 1 -- 172.21.15.112:0/4280430905 >> [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782] conn(0x7f514c073c70 msgr2=0x7f514c076090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:54.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.309+0000 7f5165432640 1 --2- 172.21.15.112:0/4280430905 >> [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782] conn(0x7f514c073c70 0x7f514c076090 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.309+0000 7f5165432640 1 -- 172.21.15.112:0/4280430905 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f514c06de80 msgr2=0x7f514c070340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:54.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.309+0000 7f5165432640 1 --2- 172.21.15.112:0/4280430905 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f514c06de80 0x7f514c070340 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5150009a60 tx=0x7f515002f9d0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.309+0000 7f5165432640 1 -- 172.21.15.112:0/4280430905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51601067e0 msgr2=0x7f5160198e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:54.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.309+0000 7f5165432640 1 --2- 172.21.15.112:0/4280430905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51601067e0 0x7f5160198e60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f514800b420 tx=0x7f514800b8e0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.309+0000 7f5165432640 1 -- 172.21.15.112:0/4280430905 shutdown_connections 2024-09-10T03:39:54.139 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.309+0000 7f5165432640 1 --2- 172.21.15.112:0/4280430905 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f514c06de80 0x7f514c070340 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.309+0000 7f5165432640 1 --2- 172.21.15.112:0/4280430905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51601067e0 0x7f5160198e60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.309+0000 7f5165432640 1 --2- 172.21.15.112:0/4280430905 >> [v2:172.21.15.112:6802/378527782,v1:172.21.15.112:6803/378527782] conn(0x7f514c073c70 0x7f514c076090 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.309+0000 7f5165432640 1 --2- 172.21.15.112:0/4280430905 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f51601007e0 0x7f5160198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.309+0000 7f5165432640 1 -- 172.21.15.112:0/4280430905 >> 172.21.15.112:0/4280430905 conn(0x7f51600fc4c0 msgr2=0x7f51600fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:54.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.309+0000 7f5165432640 1 -- 172.21.15.112:0/4280430905 shutdown_connections 2024-09-10T03:39:54.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:53.309+0000 7f5165432640 1 -- 172.21.15.112:0/4280430905 wait complete. 2024-09-10T03:39:54.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:53 smithi169 ceph-mon[35668]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:54.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:53 smithi112 ceph-mon[27737]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:54.655 INFO:teuthology.orchestra.run.smithi112.stdout:163208757255 2024-09-10T03:39:54.656 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-10T03:39:54.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.877+0000 7fb9cb5d0640 1 -- 172.21.15.112:0/525840544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9c4074e30 msgr2=0x7fb9c410dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:54.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.877+0000 7fb9cb5d0640 1 --2- 172.21.15.112:0/525840544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9c4074e30 0x7fb9c410dcb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009880 tx=0x7fb9b402a530 comp rx=0 tx=0).stop 2024-09-10T03:39:54.901 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.878+0000 7fb9cb5d0640 1 -- 172.21.15.112:0/525840544 shutdown_connections 2024-09-10T03:39:54.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.878+0000 7fb9cb5d0640 1 --2- 172.21.15.112:0/525840544 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9c4074e30 0x7fb9c410dcb0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.878+0000 7fb9cb5d0640 1 --2- 172.21.15.112:0/525840544 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9c4074510 0x7fb9c40748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.878+0000 7fb9cb5d0640 1 -- 172.21.15.112:0/525840544 >> 172.21.15.112:0/525840544 conn(0x7fb9c40fd630 msgr2=0x7fb9c40ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:54.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.878+0000 7fb9cb5d0640 1 -- 172.21.15.112:0/525840544 shutdown_connections 2024-09-10T03:39:54.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.878+0000 7fb9cb5d0640 1 -- 172.21.15.112:0/525840544 wait complete. 2024-09-10T03:39:54.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.878+0000 7fb9cb5d0640 1 Processor -- start 2024-09-10T03:39:54.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.878+0000 7fb9cb5d0640 1 -- start start 2024-09-10T03:39:54.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.878+0000 7fb9cb5d0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9c4074510 0x7fb9c419f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:54.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.878+0000 7fb9cb5d0640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9c4074e30 0x7fb9c419fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:54.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.879+0000 7fb9cb5d0640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c4111010 con 0x7fb9c4074510 2024-09-10T03:39:54.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.879+0000 7fb9cb5d0640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c4110e90 con 0x7fb9c4074e30 2024-09-10T03:39:54.902 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.879+0000 7fb9c936c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9c4074510 0x7fb9c419f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:54.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.879+0000 7fb9c936c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9c4074510 0x7fb9c419f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:42224/0 (socket says 172.21.15.112:42224) 2024-09-10T03:39:54.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.879+0000 7fb9c936c640 1 -- 172.21.15.112:0/2349040111 learned_addr learned my addr 172.21.15.112:0/2349040111 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:54.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.879+0000 7fb9c8b6b640 1 --2- 172.21.15.112:0/2349040111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9c4074e30 0x7fb9c419fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:54.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.879+0000 7fb9c936c640 1 -- 172.21.15.112:0/2349040111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9c4074e30 msgr2=0x7fb9c419fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:54.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.879+0000 7fb9c936c640 1 --2- 172.21.15.112:0/2349040111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9c4074e30 0x7fb9c419fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.879+0000 7fb9c936c640 1 -- 172.21.15.112:0/2349040111 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c4199a00 con 0x7fb9c4074510 2024-09-10T03:39:54.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.879+0000 7fb9c936c640 1 --2- 172.21.15.112:0/2349040111 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9c4074510 0x7fb9c419f840 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb9c000d6f0 tx=0x7fb9c000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:54.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.879+0000 7fb9ba7fc640 1 -- 172.21.15.112:0/2349040111 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb9c0017070 con 0x7fb9c4074510 2024-09-10T03:39:54.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.879+0000 7fb9cb5d0640 1 -- 172.21.15.112:0/2349040111 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9c4199cf0 con 0x7fb9c4074510 2024-09-10T03:39:54.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.879+0000 7fb9ba7fc640 1 -- 172.21.15.112:0/2349040111 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb9c000de00 con 0x7fb9c4074510 2024-09-10T03:39:54.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.879+0000 7fb9ba7fc640 1 -- 172.21.15.112:0/2349040111 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb9c0004bb0 con 0x7fb9c4074510 2024-09-10T03:39:54.903 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.879+0000 7fb9cb5d0640 1 -- 172.21.15.112:0/2349040111 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9c419a250 con 0x7fb9c4074510 2024-09-10T03:39:54.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.880+0000 7fb9cb5d0640 1 -- 172.21.15.112:0/2349040111 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb9c419a810 con 0x7fb9c4074510 2024-09-10T03:39:54.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.881+0000 7fb9ba7fc640 1 -- 172.21.15.112:0/2349040111 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fb9c0004e60 con 0x7fb9c4074510 2024-09-10T03:39:54.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.881+0000 7fb9ba7fc640 1 --2- 172.21.15.112:0/2349040111 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb9b006de30 0x7fb9b00702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:54.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.881+0000 7fb9ba7fc640 1 -- 172.21.15.112:0/2349040111 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7fb9c008d880 con 0x7fb9c4074510 2024-09-10T03:39:54.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.881+0000 7fb9c8b6b640 1 --2- 172.21.15.112:0/2349040111 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb9b006de30 0x7fb9b00702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:54.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.881+0000 7fb9ba7fc640 1 --2- 172.21.15.112:0/2349040111 >> [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264] conn(0x7fb9b0073c20 0x7fb9b0076040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:54.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.881+0000 7fb9ba7fc640 1 -- 172.21.15.112:0/2349040111 --> [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb9c0001d30 con 0x7fb9b0073c20 2024-09-10T03:39:54.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.881+0000 7fb9ba7fc640 1 -- 172.21.15.112:0/2349040111 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fb9c008dd20 con 0x7fb9c4074510 2024-09-10T03:39:54.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.881+0000 7fb9c9b6d640 1 --2- 172.21.15.112:0/2349040111 >> [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264] conn(0x7fb9b0073c20 0x7fb9b0076040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:54.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.882+0000 7fb9c8b6b640 1 --2- 172.21.15.112:0/2349040111 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb9b006de30 0x7fb9b00702f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb9c419b1f0 tx=0x7fb9b4031400 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:54.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.882+0000 7fb9c9b6d640 1 --2- 172.21.15.112:0/2349040111 >> [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264] conn(0x7fb9b0073c20 0x7fb9b0076040 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:54.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.882+0000 7fb9ba7fc640 1 -- 172.21.15.112:0/2349040111 <== osd.1 v2:172.21.15.112:6810/305082264 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9c0001d30 con 0x7fb9b0073c20 2024-09-10T03:39:54.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.896+0000 7fb9cb5d0640 1 -- 172.21.15.112:0/2349040111 --> [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb9c4062880 con 0x7fb9b0073c20 2024-09-10T03:39:54.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.896+0000 7fb9ba7fc640 1 -- 172.21.15.112:0/2349040111 <== osd.1 v2:172.21.15.112:6810/305082264 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb9c4062880 con 0x7fb9b0073c20 2024-09-10T03:39:54.905 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.896+0000 7fb9cb5d0640 1 -- 172.21.15.112:0/2349040111 >> [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264] conn(0x7fb9b0073c20 msgr2=0x7fb9b0076040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:54.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.896+0000 7fb9cb5d0640 1 --2- 172.21.15.112:0/2349040111 >> [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264] conn(0x7fb9b0073c20 0x7fb9b0076040 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.896+0000 7fb9cb5d0640 1 -- 172.21.15.112:0/2349040111 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb9b006de30 msgr2=0x7fb9b00702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:54.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.896+0000 7fb9cb5d0640 1 --2- 172.21.15.112:0/2349040111 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb9b006de30 0x7fb9b00702f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb9c419b1f0 tx=0x7fb9b4031400 comp rx=0 tx=0).stop 2024-09-10T03:39:54.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.896+0000 7fb9cb5d0640 1 -- 172.21.15.112:0/2349040111 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9c4074510 msgr2=0x7fb9c419f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:54.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.896+0000 7fb9cb5d0640 1 --2- 172.21.15.112:0/2349040111 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9c4074510 0x7fb9c419f840 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb9c000d6f0 tx=0x7fb9c000dbb0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.896+0000 7fb9cb5d0640 1 -- 172.21.15.112:0/2349040111 shutdown_connections 2024-09-10T03:39:54.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.896+0000 7fb9cb5d0640 1 --2- 172.21.15.112:0/2349040111 >> [v2:172.21.15.112:6810/305082264,v1:172.21.15.112:6811/305082264] conn(0x7fb9b0073c20 0x7fb9b0076040 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.896+0000 7fb9cb5d0640 1 --2- 172.21.15.112:0/2349040111 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb9b006de30 0x7fb9b00702f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.896+0000 7fb9cb5d0640 1 --2- 172.21.15.112:0/2349040111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb9c4074e30 0x7fb9c419fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.896+0000 7fb9cb5d0640 1 --2- 172.21.15.112:0/2349040111 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9c4074510 0x7fb9c419f840 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:54.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.896+0000 7fb9cb5d0640 1 -- 172.21.15.112:0/2349040111 >> 172.21.15.112:0/2349040111 conn(0x7fb9c40fd630 msgr2=0x7fb9c410b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:54.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.896+0000 7fb9cb5d0640 1 -- 172.21.15.112:0/2349040111 shutdown_connections 2024-09-10T03:39:54.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:54.896+0000 7fb9cb5d0640 1 -- 172.21.15.112:0/2349040111 wait complete. 2024-09-10T03:39:55.612 INFO:teuthology.orchestra.run.smithi112.stdout:60129542173 2024-09-10T03:39:55.613 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-10T03:39:55.966 INFO:teuthology.orchestra.run.smithi112.stdout:38654705700 2024-09-10T03:39:55.966 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-10T03:39:56.206 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:55 smithi112 ceph-mon[27737]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:56.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:55 smithi169 ceph-mon[35668]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.287+0000 7f13ef915640 1 -- 172.21.15.112:0/2536736454 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13e8102aa0 msgr2=0x7f13e8102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.287+0000 7f13ef915640 1 --2- 172.21.15.112:0/2536736454 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13e8102aa0 0x7f13e8102e80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009880 tx=0x7f13d802a530 comp rx=0 tx=0).stop 2024-09-10T03:39:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.287+0000 7f13ef915640 1 -- 172.21.15.112:0/2536736454 shutdown_connections 2024-09-10T03:39:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.287+0000 7f13ef915640 1 --2- 172.21.15.112:0/2536736454 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13e81033c0 0x7f13e810b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.287+0000 7f13ef915640 1 --2- 172.21.15.112:0/2536736454 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13e8102aa0 0x7f13e8102e80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.287+0000 7f13ef915640 1 -- 172.21.15.112:0/2536736454 >> 172.21.15.112:0/2536736454 conn(0x7f13e80fc4c0 msgr2=0x7f13e80fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.287+0000 7f13ef915640 1 -- 172.21.15.112:0/2536736454 shutdown_connections 2024-09-10T03:39:56.309 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.287+0000 7f13ef915640 1 -- 172.21.15.112:0/2536736454 wait complete. 2024-09-10T03:39:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.287+0000 7f13ef915640 1 Processor -- start 2024-09-10T03:39:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.287+0000 7f13ef915640 1 -- start start 2024-09-10T03:39:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.288+0000 7f13ef915640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13e8102aa0 0x7f13e819d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.288+0000 7f13ef915640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13e81033c0 0x7f13e819dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.288+0000 7f13ef915640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f13e810ee30 con 0x7f13e8102aa0 2024-09-10T03:39:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.288+0000 7f13ef915640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f13e810ecb0 con 0x7f13e81033c0 2024-09-10T03:39:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.288+0000 7f13ed6b1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13e8102aa0 0x7f13e819d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.288+0000 7f13ed6b1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13e8102aa0 0x7f13e819d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:59666/0 (socket says 172.21.15.112:59666) 2024-09-10T03:39:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.288+0000 7f13ed6b1640 1 -- 172.21.15.112:0/1344614246 learned_addr learned my addr 172.21.15.112:0/1344614246 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.288+0000 7f13ed6b1640 1 -- 172.21.15.112:0/1344614246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13e81033c0 msgr2=0x7f13e819dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.288+0000 7f13ed6b1640 1 --2- 172.21.15.112:0/1344614246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13e81033c0 0x7f13e819dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.288+0000 7f13ed6b1640 1 -- 172.21.15.112:0/1344614246 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e81978e0 con 0x7f13e8102aa0 2024-09-10T03:39:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.288+0000 7f13ed6b1640 1 --2- 172.21.15.112:0/1344614246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13e8102aa0 0x7f13e819d720 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f13d8031460 tx=0x7f13d80048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:56.310 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.288+0000 7f13de7fc640 1 -- 172.21.15.112:0/1344614246 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f13d803f070 con 0x7f13e8102aa0 2024-09-10T03:39:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.288+0000 7f13de7fc640 1 -- 172.21.15.112:0/1344614246 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f13d8004cf0 con 0x7f13e8102aa0 2024-09-10T03:39:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.288+0000 7f13ef915640 1 -- 172.21.15.112:0/1344614246 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e8197b70 con 0x7f13e8102aa0 2024-09-10T03:39:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.288+0000 7f13de7fc640 1 -- 172.21.15.112:0/1344614246 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f13d8035030 con 0x7f13e8102aa0 2024-09-10T03:39:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.288+0000 7f13ef915640 1 -- 172.21.15.112:0/1344614246 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e81980a0 con 0x7f13e8102aa0 2024-09-10T03:39:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.289+0000 7f13ef915640 1 -- 172.21.15.112:0/1344614246 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f13e8198640 con 0x7f13e8102aa0 2024-09-10T03:39:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.290+0000 7f13de7fc640 1 -- 172.21.15.112:0/1344614246 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f13d8042070 con 0x7f13e8102aa0 2024-09-10T03:39:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.290+0000 7f13de7fc640 1 --2- 172.21.15.112:0/1344614246 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f13d406de30 0x7f13d40702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.290+0000 7f13de7fc640 1 -- 172.21.15.112:0/1344614246 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f13d80ae330 con 0x7f13e8102aa0 2024-09-10T03:39:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.290+0000 7f13eceb0640 1 --2- 172.21.15.112:0/1344614246 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f13d406de30 0x7f13d40702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.290+0000 7f13de7fc640 1 --2- 172.21.15.112:0/1344614246 >> [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968] conn(0x7f13d4073c20 0x7f13d4076040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.290+0000 7f13de7fc640 1 -- 172.21.15.112:0/1344614246 --> [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f13d40766f0 con 0x7f13d4073c20 2024-09-10T03:39:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.290+0000 7f13de7fc640 1 -- 172.21.15.112:0/1344614246 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f13d80b5050 con 0x7f13e8102aa0 2024-09-10T03:39:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.290+0000 7f13edeb2640 1 --2- 172.21.15.112:0/1344614246 >> [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968] conn(0x7f13d4073c20 0x7f13d4076040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:56.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.291+0000 7f13eceb0640 1 --2- 172.21.15.112:0/1344614246 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f13d406de30 0x7f13d40702f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f13e80fee70 tx=0x7f13e4006d30 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:56.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.291+0000 7f13edeb2640 1 --2- 172.21.15.112:0/1344614246 >> [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968] conn(0x7f13d4073c20 0x7f13d4076040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:56.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.291+0000 7f13de7fc640 1 -- 172.21.15.112:0/1344614246 <== osd.2 v2:172.21.15.112:6818/3325040968 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f13d40766f0 con 0x7f13d4073c20 2024-09-10T03:39:56.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.305+0000 7f13ef915640 1 -- 172.21.15.112:0/1344614246 --> [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f13e80628b0 con 0x7f13d4073c20 2024-09-10T03:39:56.312 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.305+0000 7f13de7fc640 1 -- 172.21.15.112:0/1344614246 <== osd.2 v2:172.21.15.112:6818/3325040968 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f13e80628b0 con 0x7f13d4073c20 2024-09-10T03:39:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.305+0000 7f13ef915640 1 -- 172.21.15.112:0/1344614246 >> [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968] conn(0x7f13d4073c20 msgr2=0x7f13d4076040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.305+0000 7f13ef915640 1 --2- 172.21.15.112:0/1344614246 >> [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968] conn(0x7f13d4073c20 0x7f13d4076040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.305+0000 7f13ef915640 1 -- 172.21.15.112:0/1344614246 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f13d406de30 msgr2=0x7f13d40702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.305+0000 7f13ef915640 1 --2- 172.21.15.112:0/1344614246 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f13d406de30 0x7f13d40702f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f13e80fee70 tx=0x7f13e4006d30 comp rx=0 tx=0).stop 2024-09-10T03:39:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.305+0000 7f13ef915640 1 -- 172.21.15.112:0/1344614246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13e8102aa0 msgr2=0x7f13e819d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.305+0000 7f13ef915640 1 --2- 172.21.15.112:0/1344614246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13e8102aa0 0x7f13e819d720 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f13d8031460 tx=0x7f13d80048f0 comp rx=0 tx=0).stop 2024-09-10T03:39:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.305+0000 7f13ef915640 1 -- 172.21.15.112:0/1344614246 shutdown_connections 2024-09-10T03:39:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.305+0000 7f13ef915640 1 --2- 172.21.15.112:0/1344614246 >> [v2:172.21.15.112:6818/3325040968,v1:172.21.15.112:6819/3325040968] conn(0x7f13d4073c20 0x7f13d4076040 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.305+0000 7f13ef915640 1 --2- 172.21.15.112:0/1344614246 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f13d406de30 0x7f13d40702f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:56.313 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.305+0000 7f13ef915640 1 --2- 172.21.15.112:0/1344614246 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f13e81033c0 0x7f13e819dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:56.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.305+0000 7f13ef915640 1 --2- 172.21.15.112:0/1344614246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13e8102aa0 0x7f13e819d720 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:56.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.306+0000 7f13ef915640 1 -- 172.21.15.112:0/1344614246 >> 172.21.15.112:0/1344614246 conn(0x7f13e80fc4c0 msgr2=0x7f13e8108e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:56.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.306+0000 7f13ef915640 1 -- 172.21.15.112:0/1344614246 shutdown_connections 2024-09-10T03:39:56.314 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:56.306+0000 7f13ef915640 1 -- 172.21.15.112:0/1344614246 wait complete. 2024-09-10T03:39:56.929 INFO:teuthology.orchestra.run.smithi112.stdout:81604378647 2024-09-10T03:39:56.929 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-10T03:39:56.992 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:57.010 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:57.013 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:57.015 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:57.031 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:39:57.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.881+0000 7f014a178640 1 -- 172.21.15.112:0/1668870780 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0144102b00 msgr2=0x7f0144102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:57.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.881+0000 7f014a178640 1 --2- 172.21.15.112:0/1668870780 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0144102b00 0x7f0144102ee0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f01340098b0 tx=0x7f013402a560 comp rx=0 tx=0).stop 2024-09-10T03:39:57.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.881+0000 7f014a178640 1 -- 172.21.15.112:0/1668870780 shutdown_connections 2024-09-10T03:39:57.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.881+0000 7f014a178640 1 --2- 172.21.15.112:0/1668870780 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0144103420 0x7f014410b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:57.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.881+0000 7f014a178640 1 --2- 172.21.15.112:0/1668870780 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0144102b00 0x7f0144102ee0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:57.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.881+0000 7f014a178640 1 -- 172.21.15.112:0/1668870780 >> 172.21.15.112:0/1668870780 conn(0x7f01440fc4c0 msgr2=0x7f01440fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:57.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.881+0000 7f014a178640 1 -- 172.21.15.112:0/1668870780 shutdown_connections 2024-09-10T03:39:57.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.882+0000 7f014a178640 1 -- 172.21.15.112:0/1668870780 wait complete. 2024-09-10T03:39:57.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.882+0000 7f014a178640 1 Processor -- start 2024-09-10T03:39:57.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.882+0000 7f014a178640 1 -- start start 2024-09-10T03:39:57.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.882+0000 7f014a178640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0144102b00 0x7f014419d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:57.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.883+0000 7f014a178640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0144103420 0x7f014419dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:57.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.883+0000 7f014a178640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f014410ee30 con 0x7f0144102b00 2024-09-10T03:39:57.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.883+0000 7f014a178640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f014410ecb0 con 0x7f0144103420 2024-09-10T03:39:57.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.883+0000 7f01437fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0144102b00 0x7f014419d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:57.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.883+0000 7f01437fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0144102b00 0x7f014419d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:59674/0 (socket says 172.21.15.112:59674) 2024-09-10T03:39:57.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.883+0000 7f01437fe640 1 -- 172.21.15.112:0/1815272021 learned_addr learned my addr 172.21.15.112:0/1815272021 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:57.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.883+0000 7f0142ffd640 1 --2- 172.21.15.112:0/1815272021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0144103420 0x7f014419dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:57.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.883+0000 7f01437fe640 1 -- 172.21.15.112:0/1815272021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0144103420 msgr2=0x7f014419dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:57.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.883+0000 7f01437fe640 1 --2- 172.21.15.112:0/1815272021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0144103420 0x7f014419dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:57.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.883+0000 7f01437fe640 1 -- 172.21.15.112:0/1815272021 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0144197860 con 0x7f0144102b00 2024-09-10T03:39:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.883+0000 7f01437fe640 1 --2- 172.21.15.112:0/1815272021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0144102b00 0x7f014419d6a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0134031460 tx=0x7f01340048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.883+0000 7f0140ff9640 1 -- 172.21.15.112:0/1815272021 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f013403f070 con 0x7f0144102b00 2024-09-10T03:39:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.883+0000 7f014a178640 1 -- 172.21.15.112:0/1815272021 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0144197af0 con 0x7f0144102b00 2024-09-10T03:39:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.883+0000 7f0140ff9640 1 -- 172.21.15.112:0/1815272021 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f01340044d0 con 0x7f0144102b00 2024-09-10T03:39:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.883+0000 7f0140ff9640 1 -- 172.21.15.112:0/1815272021 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f0134035030 con 0x7f0144102b00 2024-09-10T03:39:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.883+0000 7f014a178640 1 -- 172.21.15.112:0/1815272021 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0144198020 con 0x7f0144102b00 2024-09-10T03:39:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.884+0000 7f014a178640 1 -- 172.21.15.112:0/1815272021 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f014406a3b0 con 0x7f0144102b00 2024-09-10T03:39:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.885+0000 7f0140ff9640 1 -- 172.21.15.112:0/1815272021 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f0134042070 con 0x7f0144102b00 2024-09-10T03:39:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.885+0000 7f0140ff9640 1 --2- 172.21.15.112:0/1815272021 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f012c06de30 0x7f012c0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.885+0000 7f0140ff9640 1 -- 172.21.15.112:0/1815272021 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f01340ae290 con 0x7f0144102b00 2024-09-10T03:39:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.885+0000 7f0142ffd640 1 --2- 172.21.15.112:0/1815272021 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f012c06de30 0x7f012c0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.886+0000 7f0142ffd640 1 --2- 172.21.15.112:0/1815272021 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f012c06de30 0x7f012c0702f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0144108940 tx=0x7f0138008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:57.890+0000 7f0140ff9640 1 -- 172.21.15.112:0/1815272021 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f013407c170 con 0x7f0144102b00 2024-09-10T03:39:58.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.061+0000 7f014a178640 1 -- 172.21.15.112:0/1815272021 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f0144105fd0 con 0x7f0144102b00 2024-09-10T03:39:58.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.061+0000 7f0140ff9640 1 -- 172.21.15.112:0/1815272021 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0134004cf0 con 0x7f0144102b00 2024-09-10T03:39:58.063 INFO:teuthology.orchestra.run.smithi112.stdout:141733920780 2024-09-10T03:39:58.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.064+0000 7f014a178640 1 -- 172.21.15.112:0/1815272021 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f012c06de30 msgr2=0x7f012c0702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:58.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.064+0000 7f014a178640 1 --2- 172.21.15.112:0/1815272021 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f012c06de30 0x7f012c0702f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0144108940 tx=0x7f0138008040 comp rx=0 tx=0).stop 2024-09-10T03:39:58.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.064+0000 7f014a178640 1 -- 172.21.15.112:0/1815272021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0144102b00 msgr2=0x7f014419d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:58.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.064+0000 7f014a178640 1 --2- 172.21.15.112:0/1815272021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0144102b00 0x7f014419d6a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0134031460 tx=0x7f01340048f0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.064+0000 7f014a178640 1 -- 172.21.15.112:0/1815272021 shutdown_connections 2024-09-10T03:39:58.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.064+0000 7f014a178640 1 --2- 172.21.15.112:0/1815272021 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f012c06de30 0x7f012c0702f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.064+0000 7f014a178640 1 --2- 172.21.15.112:0/1815272021 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0144103420 0x7f014419dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.064+0000 7f014a178640 1 --2- 172.21.15.112:0/1815272021 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0144102b00 0x7f014419d6a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.064+0000 7f014a178640 1 -- 172.21.15.112:0/1815272021 >> 172.21.15.112:0/1815272021 conn(0x7f01440fc4c0 msgr2=0x7f0144103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:58.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.064+0000 7f014a178640 1 -- 172.21.15.112:0/1815272021 shutdown_connections 2024-09-10T03:39:58.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.064+0000 7f014a178640 1 -- 172.21.15.112:0/1815272021 wait complete. 2024-09-10T03:39:58.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:57 smithi169 ceph-mon[35668]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:58.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:57 smithi112 ceph-mon[27737]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:39:58.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.537+0000 7ff3350c3640 1 -- 172.21.15.112:0/4117364684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff330106950 msgr2=0x7ff330106d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:58.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.537+0000 7ff3350c3640 1 --2- 172.21.15.112:0/4117364684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff330106950 0x7ff330106d30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff320009880 tx=0x7ff32002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.537+0000 7ff3350c3640 1 -- 172.21.15.112:0/4117364684 shutdown_connections 2024-09-10T03:39:58.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.537+0000 7ff3350c3640 1 --2- 172.21.15.112:0/4117364684 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3300ffd70 0x7ff3301001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.537+0000 7ff3350c3640 1 --2- 172.21.15.112:0/4117364684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff330106950 0x7ff330106d30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.537+0000 7ff3350c3640 1 -- 172.21.15.112:0/4117364684 >> 172.21.15.112:0/4117364684 conn(0x7ff3300fbc60 msgr2=0x7ff3300fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:58.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.537+0000 7ff3350c3640 1 -- 172.21.15.112:0/4117364684 shutdown_connections 2024-09-10T03:39:58.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.537+0000 7ff3350c3640 1 -- 172.21.15.112:0/4117364684 wait complete. 2024-09-10T03:39:58.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.537+0000 7ff3350c3640 1 Processor -- start 2024-09-10T03:39:58.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.538+0000 7ff3350c3640 1 -- start start 2024-09-10T03:39:58.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.538+0000 7ff3350c3640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3300ffd70 0x7ff33019d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:58.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.538+0000 7ff3350c3640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff330106950 0x7ff33019dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:58.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.538+0000 7ff3350c3640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff33010ef50 con 0x7ff330106950 2024-09-10T03:39:58.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.538+0000 7ff3350c3640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff33010edd0 con 0x7ff3300ffd70 2024-09-10T03:39:58.540 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.538+0000 7ff32e59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff330106950 0x7ff33019dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:58.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.538+0000 7ff32e59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff330106950 0x7ff33019dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:59692/0 (socket says 172.21.15.112:59692) 2024-09-10T03:39:58.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.538+0000 7ff32e59c640 1 -- 172.21.15.112:0/2690087251 learned_addr learned my addr 172.21.15.112:0/2690087251 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:58.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.538+0000 7ff32ed9d640 1 --2- 172.21.15.112:0/2690087251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3300ffd70 0x7ff33019d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:58.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.538+0000 7ff32e59c640 1 -- 172.21.15.112:0/2690087251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3300ffd70 msgr2=0x7ff33019d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:58.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.538+0000 7ff32e59c640 1 --2- 172.21.15.112:0/2690087251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3300ffd70 0x7ff33019d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.538+0000 7ff32e59c640 1 -- 172.21.15.112:0/2690087251 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3301978c0 con 0x7ff330106950 2024-09-10T03:39:58.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.538+0000 7ff32ed9d640 1 --2- 172.21.15.112:0/2690087251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3300ffd70 0x7ff33019d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T03:39:58.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.538+0000 7ff32e59c640 1 --2- 172.21.15.112:0/2690087251 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff330106950 0x7ff33019dc40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff32400b420 tx=0x7ff32400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:58.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.539+0000 7ff317fff640 1 -- 172.21.15.112:0/2690087251 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff324011630 con 0x7ff330106950 2024-09-10T03:39:58.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.539+0000 7ff317fff640 1 -- 172.21.15.112:0/2690087251 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff324016070 con 0x7ff330106950 2024-09-10T03:39:58.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.539+0000 7ff3350c3640 1 -- 172.21.15.112:0/2690087251 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff330197bb0 con 0x7ff330106950 2024-09-10T03:39:58.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.539+0000 7ff317fff640 1 -- 172.21.15.112:0/2690087251 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff324010d40 con 0x7ff330106950 2024-09-10T03:39:58.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.539+0000 7ff3350c3640 1 -- 172.21.15.112:0/2690087251 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff330198140 con 0x7ff330106950 2024-09-10T03:39:58.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.540+0000 7ff3350c3640 1 -- 172.21.15.112:0/2690087251 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff330103090 con 0x7ff330106950 2024-09-10T03:39:58.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.540+0000 7ff317fff640 1 -- 172.21.15.112:0/2690087251 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7ff32401b460 con 0x7ff330106950 2024-09-10T03:39:58.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.541+0000 7ff317fff640 1 --2- 172.21.15.112:0/2690087251 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff31806de30 0x7ff3180702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:58.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.541+0000 7ff317fff640 1 -- 172.21.15.112:0/2690087251 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7ff32408d0e0 con 0x7ff330106950 2024-09-10T03:39:58.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.541+0000 7ff32ed9d640 1 --2- 172.21.15.112:0/2690087251 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff31806de30 0x7ff3180702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:58.543 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.541+0000 7ff32ed9d640 1 --2- 172.21.15.112:0/2690087251 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff31806de30 0x7ff3180702f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff320006010 tx=0x7ff320004220 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:58.546 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.545+0000 7ff317fff640 1 -- 172.21.15.112:0/2690087251 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff32405afc0 con 0x7ff330106950 2024-09-10T03:39:58.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.716+0000 7ff3350c3640 1 -- 172.21.15.112:0/2690087251 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ff330198df0 con 0x7ff330106950 2024-09-10T03:39:58.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.717+0000 7ff317fff640 1 -- 172.21.15.112:0/2690087251 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff32405ec70 con 0x7ff330106950 2024-09-10T03:39:58.718 INFO:teuthology.orchestra.run.smithi112.stdout:38654705700 2024-09-10T03:39:58.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.719+0000 7ff3350c3640 1 -- 172.21.15.112:0/2690087251 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff31806de30 msgr2=0x7ff3180702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:58.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.719+0000 7ff3350c3640 1 --2- 172.21.15.112:0/2690087251 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff31806de30 0x7ff3180702f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff320006010 tx=0x7ff320004220 comp rx=0 tx=0).stop 2024-09-10T03:39:58.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.719+0000 7ff3350c3640 1 -- 172.21.15.112:0/2690087251 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff330106950 msgr2=0x7ff33019dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:58.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.719+0000 7ff3350c3640 1 --2- 172.21.15.112:0/2690087251 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff330106950 0x7ff33019dc40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff32400b420 tx=0x7ff32400b8e0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.719+0000 7ff3350c3640 1 -- 172.21.15.112:0/2690087251 shutdown_connections 2024-09-10T03:39:58.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.719+0000 7ff3350c3640 1 --2- 172.21.15.112:0/2690087251 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff31806de30 0x7ff3180702f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.719+0000 7ff3350c3640 1 --2- 172.21.15.112:0/2690087251 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff330106950 0x7ff33019dc40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.719+0000 7ff3350c3640 1 --2- 172.21.15.112:0/2690087251 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff3300ffd70 0x7ff33019d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.719+0000 7ff3350c3640 1 -- 172.21.15.112:0/2690087251 >> 172.21.15.112:0/2690087251 conn(0x7ff3300fbc60 msgr2=0x7ff3300fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:58.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.719+0000 7ff3350c3640 1 -- 172.21.15.112:0/2690087251 shutdown_connections 2024-09-10T03:39:58.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.719+0000 7ff3350c3640 1 -- 172.21.15.112:0/2690087251 wait complete. 2024-09-10T03:39:58.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.740+0000 7faeb99bb640 1 -- 172.21.15.112:0/854431648 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faeb4102aa0 msgr2=0x7faeb4102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:58.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.740+0000 7faeb99bb640 1 --2- 172.21.15.112:0/854431648 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faeb4102aa0 0x7faeb4102e80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7faea4009880 tx=0x7faea402a530 comp rx=0 tx=0).stop 2024-09-10T03:39:58.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.740+0000 7faeb99bb640 1 -- 172.21.15.112:0/854431648 shutdown_connections 2024-09-10T03:39:58.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.740+0000 7faeb99bb640 1 --2- 172.21.15.112:0/854431648 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faeb41033c0 0x7faeb410b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.740+0000 7faeb99bb640 1 --2- 172.21.15.112:0/854431648 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faeb4102aa0 0x7faeb4102e80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.740+0000 7faeb99bb640 1 -- 172.21.15.112:0/854431648 >> 172.21.15.112:0/854431648 conn(0x7faeb40fc4c0 msgr2=0x7faeb40fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:58.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.740+0000 7faeb99bb640 1 -- 172.21.15.112:0/854431648 shutdown_connections 2024-09-10T03:39:58.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.740+0000 7faeb99bb640 1 -- 172.21.15.112:0/854431648 wait complete. 2024-09-10T03:39:58.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.741+0000 7faeb99bb640 1 Processor -- start 2024-09-10T03:39:58.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.741+0000 7faeb99bb640 1 -- start start 2024-09-10T03:39:58.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.741+0000 7faeb99bb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faeb4102aa0 0x7faeb4100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:58.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.741+0000 7faeb99bb640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faeb41033c0 0x7faeb4100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:58.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.741+0000 7faeb99bb640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7faeb410edb0 con 0x7faeb41033c0 2024-09-10T03:39:58.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.741+0000 7faeb99bb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7faeb410ec30 con 0x7faeb4102aa0 2024-09-10T03:39:58.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.741+0000 7faeabfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faeb41033c0 0x7faeb4100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:58.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.741+0000 7faeabfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faeb41033c0 0x7faeb4100c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:59706/0 (socket says 172.21.15.112:59706) 2024-09-10T03:39:58.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.741+0000 7faeabfff640 1 -- 172.21.15.112:0/2463252409 learned_addr learned my addr 172.21.15.112:0/2463252409 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:39:58.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.742+0000 7faeb2ffd640 1 --2- 172.21.15.112:0/2463252409 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faeb4102aa0 0x7faeb4100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:58.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.742+0000 7faeabfff640 1 -- 172.21.15.112:0/2463252409 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faeb4102aa0 msgr2=0x7faeb4100700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:58.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.742+0000 7faeabfff640 1 --2- 172.21.15.112:0/2463252409 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faeb4102aa0 0x7faeb4100700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.742+0000 7faeabfff640 1 -- 172.21.15.112:0/2463252409 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb4101180 con 0x7faeb41033c0 2024-09-10T03:39:58.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.742+0000 7faeb2ffd640 1 --2- 172.21.15.112:0/2463252409 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faeb4102aa0 0x7faeb4100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T03:39:58.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.742+0000 7faeabfff640 1 --2- 172.21.15.112:0/2463252409 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faeb41033c0 0x7faeb4100c40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fae9c00c8c0 tx=0x7fae9c00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:58.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.742+0000 7faeb89b9640 1 -- 172.21.15.112:0/2463252409 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fae9c004020 con 0x7faeb41033c0 2024-09-10T03:39:58.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.742+0000 7faeb89b9640 1 -- 172.21.15.112:0/2463252409 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fae9c0041c0 con 0x7faeb41033c0 2024-09-10T03:39:58.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.742+0000 7faeb99bb640 1 -- 172.21.15.112:0/2463252409 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeb4101470 con 0x7faeb41033c0 2024-09-10T03:39:58.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.742+0000 7faeb89b9640 1 -- 172.21.15.112:0/2463252409 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fae9c0047a0 con 0x7faeb41033c0 2024-09-10T03:39:58.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.743+0000 7faeb99bb640 1 -- 172.21.15.112:0/2463252409 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeb41a3e00 con 0x7faeb41033c0 2024-09-10T03:39:58.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.744+0000 7faeb99bb640 1 -- 172.21.15.112:0/2463252409 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faeb406a410 con 0x7faeb41033c0 2024-09-10T03:39:58.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.744+0000 7faeb89b9640 1 -- 172.21.15.112:0/2463252409 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fae9c0079b0 con 0x7faeb41033c0 2024-09-10T03:39:58.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.745+0000 7faeb89b9640 1 --2- 172.21.15.112:0/2463252409 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7faea006de30 0x7faea00702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:39:58.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.745+0000 7faeb89b9640 1 -- 172.21.15.112:0/2463252409 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7fae9c08d7d0 con 0x7faeb41033c0 2024-09-10T03:39:58.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.745+0000 7faeb2ffd640 1 --2- 172.21.15.112:0/2463252409 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7faea006de30 0x7faea00702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:39:58.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.746+0000 7faeb2ffd640 1 --2- 172.21.15.112:0/2463252409 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7faea006de30 0x7faea00702f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7faea4031460 tx=0x7faea4009560 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:39:58.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.749+0000 7faeb89b9640 1 -- 172.21.15.112:0/2463252409 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fae9c014030 con 0x7faeb41033c0 2024-09-10T03:39:58.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.916+0000 7faeb99bb640 1 -- 172.21.15.112:0/2463252409 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7faeb4105f70 con 0x7faeb41033c0 2024-09-10T03:39:58.917 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.917+0000 7faeb89b9640 1 -- 172.21.15.112:0/2463252409 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fae9c05b610 con 0x7faeb41033c0 2024-09-10T03:39:58.918 INFO:teuthology.orchestra.run.smithi112.stdout:107374182418 2024-09-10T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.918+0000 7faeb99bb640 1 -- 172.21.15.112:0/2463252409 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7faea006de30 msgr2=0x7faea00702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.919+0000 7faeb99bb640 1 --2- 172.21.15.112:0/2463252409 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7faea006de30 0x7faea00702f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7faea4031460 tx=0x7faea4009560 comp rx=0 tx=0).stop 2024-09-10T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.919+0000 7faeb99bb640 1 -- 172.21.15.112:0/2463252409 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faeb41033c0 msgr2=0x7faeb4100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.919+0000 7faeb99bb640 1 --2- 172.21.15.112:0/2463252409 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faeb41033c0 0x7faeb4100c40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fae9c00c8c0 tx=0x7fae9c00cd80 comp rx=0 tx=0).stop 2024-09-10T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.919+0000 7faeb99bb640 1 -- 172.21.15.112:0/2463252409 shutdown_connections 2024-09-10T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.919+0000 7faeb99bb640 1 --2- 172.21.15.112:0/2463252409 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7faea006de30 0x7faea00702f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.919+0000 7faeb99bb640 1 --2- 172.21.15.112:0/2463252409 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faeb41033c0 0x7faeb4100c40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.919+0000 7faeb99bb640 1 --2- 172.21.15.112:0/2463252409 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7faeb4102aa0 0x7faeb4100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:39:58.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.919+0000 7faeb99bb640 1 -- 172.21.15.112:0/2463252409 >> 172.21.15.112:0/2463252409 conn(0x7faeb40fc4c0 msgr2=0x7faeb4108ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:39:58.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.919+0000 7faeb99bb640 1 -- 172.21.15.112:0/2463252409 shutdown_connections 2024-09-10T03:39:58.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:39:58.919+0000 7faeb99bb640 1 -- 172.21.15.112:0/2463252409 wait complete. 2024-09-10T03:39:59.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:58 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/1815272021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T03:39:59.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:58 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2690087251' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T03:39:59.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:58 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2463252409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T03:39:59.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:58 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/1815272021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T03:39:59.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:58 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/2690087251' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T03:39:59.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:58 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/2463252409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T03:39:59.582 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182417 got 107374182418 for osd.3 2024-09-10T03:39:59.582 DEBUG:teuthology.parallel:result is None 2024-09-10T03:40:00.001 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705700 got 38654705700 for osd.0 2024-09-10T03:40:00.001 DEBUG:teuthology.parallel:result is None 2024-09-10T03:40:00.208 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:39:59 smithi112 ceph-mon[27737]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:00.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.207+0000 7fa4cb568640 1 -- 172.21.15.112:0/492591377 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4c4103400 msgr2=0x7fa4c410b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:00.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.207+0000 7fa4cb568640 1 --2- 172.21.15.112:0/492591377 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4c4103400 0x7fa4c410b8f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4009880 tx=0x7fa4b402a530 comp rx=0 tx=0).stop 2024-09-10T03:40:00.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.207+0000 7fa4cb568640 1 -- 172.21.15.112:0/492591377 shutdown_connections 2024-09-10T03:40:00.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.207+0000 7fa4cb568640 1 --2- 172.21.15.112:0/492591377 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4c4103400 0x7fa4c410b8f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:00.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.207+0000 7fa4cb568640 1 --2- 172.21.15.112:0/492591377 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4c4102ae0 0x7fa4c4102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:00.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.207+0000 7fa4cb568640 1 -- 172.21.15.112:0/492591377 >> 172.21.15.112:0/492591377 conn(0x7fa4c40fc520 msgr2=0x7fa4c40fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:00.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.207+0000 7fa4cb568640 1 -- 172.21.15.112:0/492591377 shutdown_connections 2024-09-10T03:40:00.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.207+0000 7fa4cb568640 1 -- 172.21.15.112:0/492591377 wait complete. 2024-09-10T03:40:00.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.208+0000 7fa4cb568640 1 Processor -- start 2024-09-10T03:40:00.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.208+0000 7fa4cb568640 1 -- start start 2024-09-10T03:40:00.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.208+0000 7fa4cb568640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4c4102ae0 0x7fa4c4100740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:00.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.208+0000 7fa4cb568640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4c4103400 0x7fa4c4100c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:00.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.208+0000 7fa4cb568640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c410edf0 con 0x7fa4c4102ae0 2024-09-10T03:40:00.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.208+0000 7fa4cb568640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c410ec70 con 0x7fa4c4103400 2024-09-10T03:40:00.269 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.208+0000 7fa4c9304640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4c4102ae0 0x7fa4c4100740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:00.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.208+0000 7fa4c9304640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4c4102ae0 0x7fa4c4100740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:59728/0 (socket says 172.21.15.112:59728) 2024-09-10T03:40:00.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.208+0000 7fa4c9304640 1 -- 172.21.15.112:0/3328826377 learned_addr learned my addr 172.21.15.112:0/3328826377 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:00.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.208+0000 7fa4c8b03640 1 --2- 172.21.15.112:0/3328826377 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4c4103400 0x7fa4c4100c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:00.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.208+0000 7fa4c9304640 1 -- 172.21.15.112:0/3328826377 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4c4103400 msgr2=0x7fa4c4100c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:00.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.208+0000 7fa4c9304640 1 --2- 172.21.15.112:0/3328826377 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4c4103400 0x7fa4c4100c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:00.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.209+0000 7fa4c9304640 1 -- 172.21.15.112:0/3328826377 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c41011c0 con 0x7fa4c4102ae0 2024-09-10T03:40:00.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.209+0000 7fa4c9304640 1 --2- 172.21.15.112:0/3328826377 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4c4102ae0 0x7fa4c4100740 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000d8d0 tx=0x7fa4c000dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:00.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.209+0000 7fa4ba7fc640 1 -- 172.21.15.112:0/3328826377 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fa4c0017070 con 0x7fa4c4102ae0 2024-09-10T03:40:00.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.209+0000 7fa4ba7fc640 1 -- 172.21.15.112:0/3328826377 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa4c0010460 con 0x7fa4c4102ae0 2024-09-10T03:40:00.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.209+0000 7fa4cb568640 1 -- 172.21.15.112:0/3328826377 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c41014b0 con 0x7fa4c4102ae0 2024-09-10T03:40:00.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.209+0000 7fa4ba7fc640 1 -- 172.21.15.112:0/3328826377 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fa4c000f5f0 con 0x7fa4c4102ae0 2024-09-10T03:40:00.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.209+0000 7fa4cb568640 1 -- 172.21.15.112:0/3328826377 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c41a3e80 con 0x7fa4c4102ae0 2024-09-10T03:40:00.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.210+0000 7fa4cb568640 1 -- 172.21.15.112:0/3328826377 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4c406abe0 con 0x7fa4c4102ae0 2024-09-10T03:40:00.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.210+0000 7fa4ba7fc640 1 -- 172.21.15.112:0/3328826377 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fa4c0009a30 con 0x7fa4c4102ae0 2024-09-10T03:40:00.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.211+0000 7fa4ba7fc640 1 --2- 172.21.15.112:0/3328826377 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa4b006de30 0x7fa4b00702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:00.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.211+0000 7fa4ba7fc640 1 -- 172.21.15.112:0/3328826377 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7fa4c008d670 con 0x7fa4c4102ae0 2024-09-10T03:40:00.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.211+0000 7fa4c8b03640 1 --2- 172.21.15.112:0/3328826377 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa4b006de30 0x7fa4b00702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:00.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.211+0000 7fa4c8b03640 1 --2- 172.21.15.112:0/3328826377 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa4b006de30 0x7fa4b00702f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa4c41021d0 tx=0x7fa4b4031400 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:00.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.215+0000 7fa4ba7fc640 1 -- 172.21.15.112:0/3328826377 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa4c005f200 con 0x7fa4c4102ae0 2024-09-10T03:40:00.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:39:59 smithi169 ceph-mon[35668]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:00.307 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:00.390 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.389+0000 7fa4cb568640 1 -- 172.21.15.112:0/3328826377 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fa4c4102ec0 con 0x7fa4c4102ae0 2024-09-10T03:40:00.391 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.390+0000 7fa4ba7fc640 1 -- 172.21.15.112:0/3328826377 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa4c0063a30 con 0x7fa4c4102ae0 2024-09-10T03:40:00.391 INFO:teuthology.orchestra.run.smithi112.stdout:60129542174 2024-09-10T03:40:00.392 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.392+0000 7fa4cb568640 1 -- 172.21.15.112:0/3328826377 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa4b006de30 msgr2=0x7fa4b00702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:00.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.392+0000 7fa4cb568640 1 --2- 172.21.15.112:0/3328826377 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa4b006de30 0x7fa4b00702f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa4c41021d0 tx=0x7fa4b4031400 comp rx=0 tx=0).stop 2024-09-10T03:40:00.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.392+0000 7fa4cb568640 1 -- 172.21.15.112:0/3328826377 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4c4102ae0 msgr2=0x7fa4c4100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:00.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.392+0000 7fa4cb568640 1 --2- 172.21.15.112:0/3328826377 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4c4102ae0 0x7fa4c4100740 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000d8d0 tx=0x7fa4c000dd90 comp rx=0 tx=0).stop 2024-09-10T03:40:00.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.392+0000 7fa4cb568640 1 -- 172.21.15.112:0/3328826377 shutdown_connections 2024-09-10T03:40:00.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.392+0000 7fa4cb568640 1 --2- 172.21.15.112:0/3328826377 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fa4b006de30 0x7fa4b00702f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:00.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.392+0000 7fa4cb568640 1 --2- 172.21.15.112:0/3328826377 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa4c4103400 0x7fa4c4100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:00.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.392+0000 7fa4cb568640 1 --2- 172.21.15.112:0/3328826377 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa4c4102ae0 0x7fa4c4100740 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:00.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.392+0000 7fa4cb568640 1 -- 172.21.15.112:0/3328826377 >> 172.21.15.112:0/3328826377 conn(0x7fa4c40fc520 msgr2=0x7fa4c4108f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:00.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.392+0000 7fa4cb568640 1 -- 172.21.15.112:0/3328826377 shutdown_connections 2024-09-10T03:40:00.393 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:00.392+0000 7fa4cb568640 1 -- 172.21.15.112:0/3328826377 wait complete. 2024-09-10T03:40:01.049 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:00 smithi112 ceph-mon[27737]: overall HEALTH_OK 2024-09-10T03:40:01.050 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:00 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3328826377' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T03:40:01.075 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542173 got 60129542174 for osd.1 2024-09-10T03:40:01.076 DEBUG:teuthology.parallel:result is None 2024-09-10T03:40:01.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.237+0000 7f542f9f7640 1 -- 172.21.15.112:0/4024508312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54281007e0 msgr2=0x7f5428100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:01.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.237+0000 7f542f9f7640 1 --2- 172.21.15.112:0/4024508312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54281007e0 0x7f5428100c40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5418009880 tx=0x7f541802a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:01.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.237+0000 7f542f9f7640 1 -- 172.21.15.112:0/4024508312 shutdown_connections 2024-09-10T03:40:01.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.237+0000 7f542f9f7640 1 --2- 172.21.15.112:0/4024508312 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54281007e0 0x7f5428100c40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:01.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.237+0000 7f542f9f7640 1 --2- 172.21.15.112:0/4024508312 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54281067e0 0x7f5428106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:01.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.237+0000 7f542f9f7640 1 -- 172.21.15.112:0/4024508312 >> 172.21.15.112:0/4024508312 conn(0x7f54280fc4c0 msgr2=0x7f54280fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:01.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.237+0000 7f542f9f7640 1 -- 172.21.15.112:0/4024508312 shutdown_connections 2024-09-10T03:40:01.239 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.237+0000 7f542f9f7640 1 -- 172.21.15.112:0/4024508312 wait complete. 2024-09-10T03:40:01.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.238+0000 7f542f9f7640 1 Processor -- start 2024-09-10T03:40:01.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.238+0000 7f542f9f7640 1 -- start start 2024-09-10T03:40:01.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.238+0000 7f542f9f7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54281007e0 0x7f542819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:01.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.238+0000 7f542f9f7640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54281067e0 0x7f542819cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:01.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.238+0000 7f542f9f7640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f542810dd10 con 0x7f54281067e0 2024-09-10T03:40:01.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.238+0000 7f542f9f7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f542810db90 con 0x7f54281007e0 2024-09-10T03:40:01.240 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.238+0000 7f542cf92640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54281067e0 0x7f542819cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:01.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.238+0000 7f542cf92640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54281067e0 0x7f542819cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:59752/0 (socket says 172.21.15.112:59752) 2024-09-10T03:40:01.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.238+0000 7f542cf92640 1 -- 172.21.15.112:0/1722875889 learned_addr learned my addr 172.21.15.112:0/1722875889 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:01.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.238+0000 7f542d793640 1 --2- 172.21.15.112:0/1722875889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54281007e0 0x7f542819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:01.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.238+0000 7f542cf92640 1 -- 172.21.15.112:0/1722875889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54281007e0 msgr2=0x7f542819c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:01.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.238+0000 7f542cf92640 1 --2- 172.21.15.112:0/1722875889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54281007e0 0x7f542819c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:01.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.239+0000 7f542cf92640 1 -- 172.21.15.112:0/1722875889 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5428196700 con 0x7f54281067e0 2024-09-10T03:40:01.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.239+0000 7f542cf92640 1 --2- 172.21.15.112:0/1722875889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54281067e0 0x7f542819cb70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5418031460 tx=0x7f54180048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:01.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.239+0000 7f541e7fc640 1 -- 172.21.15.112:0/1722875889 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f541803f070 con 0x7f54281067e0 2024-09-10T03:40:01.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.239+0000 7f541e7fc640 1 -- 172.21.15.112:0/1722875889 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5418004cf0 con 0x7f54281067e0 2024-09-10T03:40:01.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.239+0000 7f542f9f7640 1 -- 172.21.15.112:0/1722875889 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5428196990 con 0x7f54281067e0 2024-09-10T03:40:01.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.239+0000 7f541e7fc640 1 -- 172.21.15.112:0/1722875889 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f5418035030 con 0x7f54281067e0 2024-09-10T03:40:01.241 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.239+0000 7f542f9f7640 1 -- 172.21.15.112:0/1722875889 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5428196e90 con 0x7f54281067e0 2024-09-10T03:40:01.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.240+0000 7f542f9f7640 1 -- 172.21.15.112:0/1722875889 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5428101f20 con 0x7f54281067e0 2024-09-10T03:40:01.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.240+0000 7f541e7fc640 1 -- 172.21.15.112:0/1722875889 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f5418042070 con 0x7f54281067e0 2024-09-10T03:40:01.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.241+0000 7f541e7fc640 1 --2- 172.21.15.112:0/1722875889 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f541406de30 0x7f54140702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:01.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.241+0000 7f541e7fc640 1 -- 172.21.15.112:0/1722875889 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f54180ae330 con 0x7f54281067e0 2024-09-10T03:40:01.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.241+0000 7f542d793640 1 --2- 172.21.15.112:0/1722875889 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f541406de30 0x7f54140702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:01.242 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.241+0000 7f542d793640 1 --2- 172.21.15.112:0/1722875889 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f541406de30 0x7f54140702f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f542400a810 tx=0x7f5424008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:01.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.246+0000 7f541e7fc640 1 -- 172.21.15.112:0/1722875889 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f541807c210 con 0x7f54281067e0 2024-09-10T03:40:01.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:00 smithi169 ceph-mon[35668]: overall HEALTH_OK 2024-09-10T03:40:01.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:00 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3328826377' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T03:40:01.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.421+0000 7f542f9f7640 1 -- 172.21.15.112:0/1722875889 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f5428108d70 con 0x7f54281067e0 2024-09-10T03:40:01.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.422+0000 7f541e7fc640 1 -- 172.21.15.112:0/1722875889 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f54180ae150 con 0x7f54281067e0 2024-09-10T03:40:01.423 INFO:teuthology.orchestra.run.smithi112.stdout:163208757257 2024-09-10T03:40:01.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.424+0000 7f542f9f7640 1 -- 172.21.15.112:0/1722875889 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f541406de30 msgr2=0x7f54140702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:01.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.424+0000 7f542f9f7640 1 --2- 172.21.15.112:0/1722875889 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f541406de30 0x7f54140702f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f542400a810 tx=0x7f5424008040 comp rx=0 tx=0).stop 2024-09-10T03:40:01.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.424+0000 7f542f9f7640 1 -- 172.21.15.112:0/1722875889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54281067e0 msgr2=0x7f542819cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:01.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.424+0000 7f542f9f7640 1 --2- 172.21.15.112:0/1722875889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54281067e0 0x7f542819cb70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5418031460 tx=0x7f54180048f0 comp rx=0 tx=0).stop 2024-09-10T03:40:01.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.424+0000 7f542f9f7640 1 -- 172.21.15.112:0/1722875889 shutdown_connections 2024-09-10T03:40:01.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.424+0000 7f542f9f7640 1 --2- 172.21.15.112:0/1722875889 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f541406de30 0x7f54140702f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:01.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.424+0000 7f542f9f7640 1 --2- 172.21.15.112:0/1722875889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f54281067e0 0x7f542819cb70 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:01.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.424+0000 7f542f9f7640 1 --2- 172.21.15.112:0/1722875889 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f54281007e0 0x7f542819c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:01.425 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.424+0000 7f542f9f7640 1 -- 172.21.15.112:0/1722875889 >> 172.21.15.112:0/1722875889 conn(0x7f54280fc4c0 msgr2=0x7f54280fcdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:01.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.424+0000 7f542f9f7640 1 -- 172.21.15.112:0/1722875889 shutdown_connections 2024-09-10T03:40:01.426 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:01.424+0000 7f542f9f7640 1 -- 172.21.15.112:0/1722875889 wait complete. 2024-09-10T03:40:02.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:01 smithi112 ceph-mon[27737]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:02.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:01 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/1722875889' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T03:40:02.136 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757255 got 163208757257 for osd.5 2024-09-10T03:40:02.136 DEBUG:teuthology.parallel:result is None 2024-09-10T03:40:02.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:01 smithi169 ceph-mon[35668]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:02.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:01 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/1722875889' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T03:40:02.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.303+0000 7f559a1b1640 1 -- 172.21.15.112:0/2520338479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5594100820 msgr2=0x7f5594100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:02.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.303+0000 7f559a1b1640 1 --2- 172.21.15.112:0/2520338479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5594100820 0x7f5594100c80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5588009880 tx=0x7f558802a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:02.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.303+0000 7f559a1b1640 1 -- 172.21.15.112:0/2520338479 shutdown_connections 2024-09-10T03:40:02.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.303+0000 7f559a1b1640 1 --2- 172.21.15.112:0/2520338479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5594100820 0x7f5594100c80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:02.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.303+0000 7f559a1b1640 1 --2- 172.21.15.112:0/2520338479 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5594106820 0x7f5594106c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:02.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.303+0000 7f559a1b1640 1 -- 172.21.15.112:0/2520338479 >> 172.21.15.112:0/2520338479 conn(0x7f55940fc520 msgr2=0x7f55940fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:02.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.303+0000 7f559a1b1640 1 -- 172.21.15.112:0/2520338479 shutdown_connections 2024-09-10T03:40:02.304 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.303+0000 7f559a1b1640 1 -- 172.21.15.112:0/2520338479 wait complete. 2024-09-10T03:40:02.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.303+0000 7f559a1b1640 1 Processor -- start 2024-09-10T03:40:02.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.304+0000 7f559a1b1640 1 -- start start 2024-09-10T03:40:02.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.304+0000 7f559a1b1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5594100820 0x7f559419d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:02.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.304+0000 7f559a1b1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5594106820 0x7f559419dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:02.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.304+0000 7f559a1b1640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f559410dd50 con 0x7f5594106820 2024-09-10T03:40:02.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.304+0000 7f559a1b1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f559410dbd0 con 0x7f5594100820 2024-09-10T03:40:02.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.304+0000 7f5592ffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5594106820 0x7f559419dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:02.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.304+0000 7f5592ffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5594106820 0x7f559419dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:59774/0 (socket says 172.21.15.112:59774) 2024-09-10T03:40:02.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.304+0000 7f5592ffd640 1 -- 172.21.15.112:0/2581526093 learned_addr learned my addr 172.21.15.112:0/2581526093 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:02.306 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.304+0000 7f55937fe640 1 --2- 172.21.15.112:0/2581526093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5594100820 0x7f559419d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:02.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.304+0000 7f5592ffd640 1 -- 172.21.15.112:0/2581526093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5594100820 msgr2=0x7f559419d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:02.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.304+0000 7f5592ffd640 1 --2- 172.21.15.112:0/2581526093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5594100820 0x7f559419d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:02.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.304+0000 7f5592ffd640 1 -- 172.21.15.112:0/2581526093 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5594197840 con 0x7f5594106820 2024-09-10T03:40:02.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.304+0000 7f55937fe640 1 --2- 172.21.15.112:0/2581526093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5594100820 0x7f559419d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T03:40:02.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.304+0000 7f5592ffd640 1 --2- 172.21.15.112:0/2581526093 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5594106820 0x7f559419dc00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5588031460 tx=0x7f55880048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:02.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.304+0000 7f5590ff9640 1 -- 172.21.15.112:0/2581526093 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f558803f070 con 0x7f5594106820 2024-09-10T03:40:02.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.305+0000 7f5590ff9640 1 -- 172.21.15.112:0/2581526093 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5588004cf0 con 0x7f5594106820 2024-09-10T03:40:02.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.305+0000 7f559a1b1640 1 -- 172.21.15.112:0/2581526093 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5594197ad0 con 0x7f5594106820 2024-09-10T03:40:02.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.305+0000 7f5590ff9640 1 -- 172.21.15.112:0/2581526093 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f5588035030 con 0x7f5594106820 2024-09-10T03:40:02.307 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.305+0000 7f559a1b1640 1 -- 172.21.15.112:0/2581526093 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5594198000 con 0x7f5594106820 2024-09-10T03:40:02.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.306+0000 7f559a1b1640 1 -- 172.21.15.112:0/2581526093 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5594101f60 con 0x7f5594106820 2024-09-10T03:40:02.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.306+0000 7f5590ff9640 1 -- 172.21.15.112:0/2581526093 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f5588042070 con 0x7f5594106820 2024-09-10T03:40:02.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.307+0000 7f5590ff9640 1 --2- 172.21.15.112:0/2581526093 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f557c06de90 0x7f557c070350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:02.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.307+0000 7f55937fe640 1 --2- 172.21.15.112:0/2581526093 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f557c06de90 0x7f557c070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:02.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.307+0000 7f5590ff9640 1 -- 172.21.15.112:0/2581526093 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f55880ae260 con 0x7f5594106820 2024-09-10T03:40:02.308 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.307+0000 7f55937fe640 1 --2- 172.21.15.112:0/2581526093 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f557c06de90 0x7f557c070350 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5584006fd0 tx=0x7f5584008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:02.311 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.311+0000 7f5590ff9640 1 -- 172.21.15.112:0/2581526093 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f558807c1c0 con 0x7f5594106820 2024-09-10T03:40:02.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.479+0000 7f559a1b1640 1 -- 172.21.15.112:0/2581526093 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f5594198370 con 0x7f5594106820 2024-09-10T03:40:02.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.480+0000 7f5590ff9640 1 -- 172.21.15.112:0/2581526093 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f558807fe70 con 0x7f5594106820 2024-09-10T03:40:02.481 INFO:teuthology.orchestra.run.smithi112.stdout:81604378648 2024-09-10T03:40:02.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.482+0000 7f559a1b1640 1 -- 172.21.15.112:0/2581526093 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f557c06de90 msgr2=0x7f557c070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:02.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.482+0000 7f559a1b1640 1 --2- 172.21.15.112:0/2581526093 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f557c06de90 0x7f557c070350 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5584006fd0 tx=0x7f5584008040 comp rx=0 tx=0).stop 2024-09-10T03:40:02.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.482+0000 7f559a1b1640 1 -- 172.21.15.112:0/2581526093 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5594106820 msgr2=0x7f559419dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:02.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.482+0000 7f559a1b1640 1 --2- 172.21.15.112:0/2581526093 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5594106820 0x7f559419dc00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5588031460 tx=0x7f55880048f0 comp rx=0 tx=0).stop 2024-09-10T03:40:02.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.482+0000 7f559a1b1640 1 -- 172.21.15.112:0/2581526093 shutdown_connections 2024-09-10T03:40:02.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.482+0000 7f559a1b1640 1 --2- 172.21.15.112:0/2581526093 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f557c06de90 0x7f557c070350 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:02.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.482+0000 7f559a1b1640 1 --2- 172.21.15.112:0/2581526093 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5594106820 0x7f559419dc00 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:02.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.482+0000 7f559a1b1640 1 --2- 172.21.15.112:0/2581526093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5594100820 0x7f559419d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:02.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.482+0000 7f559a1b1640 1 -- 172.21.15.112:0/2581526093 >> 172.21.15.112:0/2581526093 conn(0x7f55940fc520 msgr2=0x7f5594104630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:02.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.482+0000 7f559a1b1640 1 -- 172.21.15.112:0/2581526093 shutdown_connections 2024-09-10T03:40:02.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:02.482+0000 7f559a1b1640 1 -- 172.21.15.112:0/2581526093 wait complete. 2024-09-10T03:40:03.035 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920779 got 141733920780 for osd.4 2024-09-10T03:40:03.035 DEBUG:teuthology.parallel:result is None 2024-09-10T03:40:03.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:02 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/2581526093' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T03:40:03.294 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:02 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2581526093' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T03:40:03.725 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378647 got 81604378648 for osd.2 2024-09-10T03:40:03.725 DEBUG:teuthology.parallel:result is None 2024-09-10T03:40:03.726 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-10T03:40:03.726 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T03:40:03.991 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:04.247 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:03 smithi112 ceph-mon[27737]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:04.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:03 smithi169 ceph-mon[35668]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:04.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.465+0000 7f2dbf555640 1 -- 172.21.15.112:0/1306216134 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2db8106820 msgr2=0x7f2db8106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:04.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.465+0000 7f2dbf555640 1 --2- 172.21.15.112:0/1306216134 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2db8106820 0x7f2db8106c00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2db4009880 tx=0x7f2db402a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:04.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.465+0000 7f2dbf555640 1 -- 172.21.15.112:0/1306216134 shutdown_connections 2024-09-10T03:40:04.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.465+0000 7f2dbf555640 1 --2- 172.21.15.112:0/1306216134 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2db8100820 0x7f2db8100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:04.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.465+0000 7f2dbf555640 1 --2- 172.21.15.112:0/1306216134 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2db8106820 0x7f2db8106c00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:04.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.465+0000 7f2dbf555640 1 -- 172.21.15.112:0/1306216134 >> 172.21.15.112:0/1306216134 conn(0x7f2db80fc520 msgr2=0x7f2db80fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:04.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.465+0000 7f2dbf555640 1 -- 172.21.15.112:0/1306216134 shutdown_connections 2024-09-10T03:40:04.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.465+0000 7f2dbf555640 1 -- 172.21.15.112:0/1306216134 wait complete. 2024-09-10T03:40:04.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.466+0000 7f2dbf555640 1 Processor -- start 2024-09-10T03:40:04.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.466+0000 7f2dbf555640 1 -- start start 2024-09-10T03:40:04.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.466+0000 7f2dbf555640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2db8100820 0x7f2db819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:04.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.466+0000 7f2dbf555640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2db8106820 0x7f2db819cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:04.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.466+0000 7f2dbf555640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2db810dd50 con 0x7f2db8106820 2024-09-10T03:40:04.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.466+0000 7f2dbf555640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2db810dbd0 con 0x7f2db8100820 2024-09-10T03:40:04.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.466+0000 7f2dbcaf0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2db8106820 0x7f2db819cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:04.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.466+0000 7f2dbcaf0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2db8106820 0x7f2db819cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:59794/0 (socket says 172.21.15.112:59794) 2024-09-10T03:40:04.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.466+0000 7f2dbcaf0640 1 -- 172.21.15.112:0/496857870 learned_addr learned my addr 172.21.15.112:0/496857870 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:04.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.466+0000 7f2dbd2f1640 1 --2- 172.21.15.112:0/496857870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2db8100820 0x7f2db819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:04.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.466+0000 7f2dbcaf0640 1 -- 172.21.15.112:0/496857870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2db8100820 msgr2=0x7f2db819c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:04.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.466+0000 7f2dbcaf0640 1 --2- 172.21.15.112:0/496857870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2db8100820 0x7f2db819c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:04.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.466+0000 7f2dbcaf0640 1 -- 172.21.15.112:0/496857870 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db8196720 con 0x7f2db8106820 2024-09-10T03:40:04.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.466+0000 7f2dbcaf0640 1 --2- 172.21.15.112:0/496857870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2db8106820 0x7f2db819cb90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2da800d910 tx=0x7f2da800ddd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:04.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.467+0000 7f2dae7fc640 1 -- 172.21.15.112:0/496857870 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2da801b070 con 0x7f2db8106820 2024-09-10T03:40:04.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.467+0000 7f2dbf555640 1 -- 172.21.15.112:0/496857870 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db81969b0 con 0x7f2db8106820 2024-09-10T03:40:04.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.467+0000 7f2dae7fc640 1 -- 172.21.15.112:0/496857870 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2da800eb70 con 0x7f2db8106820 2024-09-10T03:40:04.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.467+0000 7f2dae7fc640 1 -- 172.21.15.112:0/496857870 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2da8014600 con 0x7f2db8106820 2024-09-10T03:40:04.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.467+0000 7f2dbf555640 1 -- 172.21.15.112:0/496857870 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db8196f10 con 0x7f2db8106820 2024-09-10T03:40:04.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.467+0000 7f2dbf555640 1 -- 172.21.15.112:0/496857870 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2db8101f60 con 0x7f2db8106820 2024-09-10T03:40:04.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.468+0000 7f2dae7fc640 1 -- 172.21.15.112:0/496857870 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f2da801e070 con 0x7f2db8106820 2024-09-10T03:40:04.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.468+0000 7f2dae7fc640 1 --2- 172.21.15.112:0/496857870 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2da406de30 0x7f2da40702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:04.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.469+0000 7f2dae7fc640 1 -- 172.21.15.112:0/496857870 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f2da808f0b0 con 0x7f2db8106820 2024-09-10T03:40:04.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.469+0000 7f2dbd2f1640 1 --2- 172.21.15.112:0/496857870 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2da406de30 0x7f2da40702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:04.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.469+0000 7f2dbd2f1640 1 --2- 172.21.15.112:0/496857870 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2da406de30 0x7f2da40702f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2db40096c0 tx=0x7f2db4031400 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:04.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.473+0000 7f2dae7fc640 1 -- 172.21.15.112:0/496857870 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2da8060c40 con 0x7f2db8106820 2024-09-10T03:40:04.639 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.638+0000 7f2dbf555640 1 -- 172.21.15.112:0/496857870 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2db8197220 con 0x7f2da406de30 2024-09-10T03:40:04.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.639+0000 7f2dae7fc640 1 -- 172.21.15.112:0/496857870 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+12321 (secure 0 0 0) 0x7f2db8197220 con 0x7f2da406de30 2024-09-10T03:40:04.640 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:40:04.640 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-10T03:40:04.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.641+0000 7f2dbf555640 1 -- 172.21.15.112:0/496857870 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2da406de30 msgr2=0x7f2da40702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:04.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.641+0000 7f2dbf555640 1 --2- 172.21.15.112:0/496857870 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2da406de30 0x7f2da40702f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2db40096c0 tx=0x7f2db4031400 comp rx=0 tx=0).stop 2024-09-10T03:40:04.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.641+0000 7f2dbf555640 1 -- 172.21.15.112:0/496857870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2db8106820 msgr2=0x7f2db819cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:04.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.641+0000 7f2dbf555640 1 --2- 172.21.15.112:0/496857870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2db8106820 0x7f2db819cb90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2da800d910 tx=0x7f2da800ddd0 comp rx=0 tx=0).stop 2024-09-10T03:40:04.642 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.641+0000 7f2dbf555640 1 -- 172.21.15.112:0/496857870 shutdown_connections 2024-09-10T03:40:04.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.641+0000 7f2dbf555640 1 --2- 172.21.15.112:0/496857870 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2da406de30 0x7f2da40702f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:04.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.641+0000 7f2dbf555640 1 --2- 172.21.15.112:0/496857870 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2db8106820 0x7f2db819cb90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:04.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.641+0000 7f2dbf555640 1 --2- 172.21.15.112:0/496857870 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2db8100820 0x7f2db819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:04.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.641+0000 7f2dbf555640 1 -- 172.21.15.112:0/496857870 >> 172.21.15.112:0/496857870 conn(0x7f2db80fc520 msgr2=0x7f2db80fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:04.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.641+0000 7f2dbf555640 1 -- 172.21.15.112:0/496857870 shutdown_connections 2024-09-10T03:40:04.643 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:04.641+0000 7f2dbf555640 1 -- 172.21.15.112:0/496857870 wait complete. 2024-09-10T03:40:05.274 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:04 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:40:05.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:04 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:40:05.302 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-10T03:40:02.862875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":182656,"kb_used_data":3828,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":562288256,"statfs":{"total":575970213888,"available":575783174144,"internally_reserved":0,"allocated":3919872,"data_stored":2589960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":182966945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001913"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":192,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-10T03:39:36.570327+0000","last_change":"2024-09-10T03:39:08.104142+0000","last_active":"2024-09-10T03:39:36.570327+0000","last_peered":"2024-09-10T03:39:36.570327+0000","last_clean":"2024-09-10T03:39:36.570327+0000","last_became_active":"2024-09-10T03:39:08.103780+0000","last_became_peered":"2024-09-10T03:39:08.103780+0000","last_unstale":"2024-09-10T03:39:36.570327+0000","last_undegraded":"2024-09-10T03:39:36.570327+0000","last_fullsized":"2024-09-10T03:39:36.570327+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T03:38:15.123910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T03:38:15.123910+0000","last_clean_scrub_stamp":"2024-09-10T03:38:15.123910+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:24.883070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":38,"seq":163208757257,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31536,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713616,"statfs":{"total":95995035648,"available":95962742784,"internally_reserved":0,"allocated":356352,"data_stored":136476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920781,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31540,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713612,"statfs":{"total":95995035648,"available":95962738688,"internally_reserved":0,"allocated":356352,"data_stored":136476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":950272,"data_stored":726844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378648,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27384,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717768,"statfs":{"total":95995035648,"available":95966994432,"internally_reserved":0,"allocated":356352,"data_stored":136476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542175,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":950272,"data_stored":726844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705702,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":950272,"data_stored":726844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T03:40:05.303 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T03:40:05.549 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:05.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.968+0000 7f62c584d640 1 -- 172.21.15.112:0/1430984608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62c01033c0 msgr2=0x7f62c010b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:05.970 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.968+0000 7f62c584d640 1 --2- 172.21.15.112:0/1430984608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62c01033c0 0x7f62c010b8b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f62a8009880 tx=0x7f62a802a530 comp rx=0 tx=0).stop 2024-09-10T03:40:05.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.968+0000 7f62c584d640 1 -- 172.21.15.112:0/1430984608 shutdown_connections 2024-09-10T03:40:05.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.968+0000 7f62c584d640 1 --2- 172.21.15.112:0/1430984608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62c01033c0 0x7f62c010b8b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:05.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.968+0000 7f62c584d640 1 --2- 172.21.15.112:0/1430984608 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c0102aa0 0x7f62c0102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:05.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.968+0000 7f62c584d640 1 -- 172.21.15.112:0/1430984608 >> 172.21.15.112:0/1430984608 conn(0x7f62c00fc4c0 msgr2=0x7f62c00fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:05.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.969+0000 7f62c584d640 1 -- 172.21.15.112:0/1430984608 shutdown_connections 2024-09-10T03:40:05.971 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.969+0000 7f62c584d640 1 -- 172.21.15.112:0/1430984608 wait complete. 2024-09-10T03:40:05.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.969+0000 7f62c584d640 1 Processor -- start 2024-09-10T03:40:05.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.970+0000 7f62c584d640 1 -- start start 2024-09-10T03:40:05.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.970+0000 7f62c584d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c0102aa0 0x7f62c0100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:05.972 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.970+0000 7f62c584d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62c01033c0 0x7f62c0100c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:05.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.970+0000 7f62c584d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f62c010edd0 con 0x7f62c01033c0 2024-09-10T03:40:05.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.970+0000 7f62c584d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f62c010ec50 con 0x7f62c0102aa0 2024-09-10T03:40:05.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.970+0000 7f62beffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c0102aa0 0x7f62c0100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:05.973 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.970+0000 7f62beffd640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c0102aa0 0x7f62c0100700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.112:50474/0 (socket says 172.21.15.112:50474) 2024-09-10T03:40:05.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.970+0000 7f62beffd640 1 -- 172.21.15.112:0/12770470 learned_addr learned my addr 172.21.15.112:0/12770470 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:05.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.970+0000 7f62be7fc640 1 --2- 172.21.15.112:0/12770470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62c01033c0 0x7f62c0100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:05.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.970+0000 7f62be7fc640 1 -- 172.21.15.112:0/12770470 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c0102aa0 msgr2=0x7f62c0100700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:05.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.970+0000 7f62be7fc640 1 --2- 172.21.15.112:0/12770470 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c0102aa0 0x7f62c0100700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:05.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.970+0000 7f62be7fc640 1 -- 172.21.15.112:0/12770470 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62c0102480 con 0x7f62c01033c0 2024-09-10T03:40:05.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.970+0000 7f62beffd640 1 --2- 172.21.15.112:0/12770470 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c0102aa0 0x7f62c0100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T03:40:05.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.971+0000 7f62be7fc640 1 --2- 172.21.15.112:0/12770470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62c01033c0 0x7f62c0100c40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f62a802aa50 tx=0x7f62a80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:05.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.971+0000 7f62c484b640 1 -- 172.21.15.112:0/12770470 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f62a803f070 con 0x7f62c01033c0 2024-09-10T03:40:05.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.971+0000 7f62c584d640 1 -- 172.21.15.112:0/12770470 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62c0101180 con 0x7f62c01033c0 2024-09-10T03:40:05.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.971+0000 7f62c484b640 1 -- 172.21.15.112:0/12770470 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f62a80035f0 con 0x7f62c01033c0 2024-09-10T03:40:05.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.971+0000 7f62c484b640 1 -- 172.21.15.112:0/12770470 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f62a8035030 con 0x7f62c01033c0 2024-09-10T03:40:05.974 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.971+0000 7f62c584d640 1 -- 172.21.15.112:0/12770470 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62c019bb40 con 0x7f62c01033c0 2024-09-10T03:40:05.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.972+0000 7f62c584d640 1 -- 172.21.15.112:0/12770470 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62c0102650 con 0x7f62c01033c0 2024-09-10T03:40:05.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.972+0000 7f62c484b640 1 -- 172.21.15.112:0/12770470 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f62a8042070 con 0x7f62c01033c0 2024-09-10T03:40:05.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.973+0000 7f62c484b640 1 --2- 172.21.15.112:0/12770470 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f62ac06de30 0x7f62ac0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:05.975 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.973+0000 7f62beffd640 1 --2- 172.21.15.112:0/12770470 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f62ac06de30 0x7f62ac0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:05.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.973+0000 7f62c484b640 1 -- 172.21.15.112:0/12770470 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f62a80ae050 con 0x7f62c01033c0 2024-09-10T03:40:05.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.974+0000 7f62beffd640 1 --2- 172.21.15.112:0/12770470 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f62ac06de30 0x7f62ac0702f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009fd0 tx=0x7f62b0009700 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:05.978 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:05.977+0000 7f62c484b640 1 -- 172.21.15.112:0/12770470 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f62a807fbe0 con 0x7f62c01033c0 2024-09-10T03:40:06.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:06.139+0000 7f62c584d640 1 -- 172.21.15.112:0/12770470 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f62c0101c30 con 0x7f62ac06de30 2024-09-10T03:40:06.140 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:06.139+0000 7f62c484b640 1 -- 172.21.15.112:0/12770470 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+12321 (secure 0 0 0) 0x7f62c0101c30 con 0x7f62ac06de30 2024-09-10T03:40:06.141 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:40:06.141 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-10T03:40:06.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:06.141+0000 7f62c584d640 1 -- 172.21.15.112:0/12770470 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f62ac06de30 msgr2=0x7f62ac0702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:06.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:06.141+0000 7f62c584d640 1 --2- 172.21.15.112:0/12770470 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f62ac06de30 0x7f62ac0702f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009fd0 tx=0x7f62b0009700 comp rx=0 tx=0).stop 2024-09-10T03:40:06.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:06.141+0000 7f62c584d640 1 -- 172.21.15.112:0/12770470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62c01033c0 msgr2=0x7f62c0100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:06.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:06.141+0000 7f62c584d640 1 --2- 172.21.15.112:0/12770470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62c01033c0 0x7f62c0100c40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f62a802aa50 tx=0x7f62a80045e0 comp rx=0 tx=0).stop 2024-09-10T03:40:06.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:06.142+0000 7f62c584d640 1 -- 172.21.15.112:0/12770470 shutdown_connections 2024-09-10T03:40:06.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:06.142+0000 7f62c584d640 1 --2- 172.21.15.112:0/12770470 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f62ac06de30 0x7f62ac0702f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:06.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:06.142+0000 7f62c584d640 1 --2- 172.21.15.112:0/12770470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f62c01033c0 0x7f62c0100c40 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:06.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:06.142+0000 7f62c584d640 1 --2- 172.21.15.112:0/12770470 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f62c0102aa0 0x7f62c0100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:06.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:06.142+0000 7f62c584d640 1 -- 172.21.15.112:0/12770470 >> 172.21.15.112:0/12770470 conn(0x7f62c00fc4c0 msgr2=0x7f62c0108ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:06.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:06.142+0000 7f62c584d640 1 -- 172.21.15.112:0/12770470 shutdown_connections 2024-09-10T03:40:06.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:06.142+0000 7f62c584d640 1 -- 172.21.15.112:0/12770470 wait complete. 2024-09-10T03:40:06.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:05 smithi169 ceph-mon[35668]: from='client.14524 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T03:40:06.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:05 smithi169 ceph-mon[35668]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:06.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:05 smithi112 ceph-mon[27737]: from='client.14524 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T03:40:06.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:05 smithi112 ceph-mon[27737]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:06.774 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-09-10T03:40:04.864199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":182656,"kb_used_data":3828,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":562288256,"statfs":{"total":575970213888,"available":575783174144,"internally_reserved":0,"allocated":3919872,"data_stored":2589960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":182966945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002941"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":192,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-10T03:39:36.570327+0000","last_change":"2024-09-10T03:39:08.104142+0000","last_active":"2024-09-10T03:39:36.570327+0000","last_peered":"2024-09-10T03:39:36.570327+0000","last_clean":"2024-09-10T03:39:36.570327+0000","last_became_active":"2024-09-10T03:39:08.103780+0000","last_became_peered":"2024-09-10T03:39:08.103780+0000","last_unstale":"2024-09-10T03:39:36.570327+0000","last_undegraded":"2024-09-10T03:39:36.570327+0000","last_fullsized":"2024-09-10T03:39:36.570327+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T03:38:15.123910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T03:38:15.123910+0000","last_clean_scrub_stamp":"2024-09-10T03:38:15.123910+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T08:19:24.883070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":38,"seq":163208757257,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31536,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713616,"statfs":{"total":95995035648,"available":95962742784,"internally_reserved":0,"allocated":356352,"data_stored":136476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920782,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31540,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713612,"statfs":{"total":95995035648,"available":95962738688,"internally_reserved":0,"allocated":356352,"data_stored":136476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182420,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":950272,"data_stored":726844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378648,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27384,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717768,"statfs":{"total":95995035648,"available":95966994432,"internally_reserved":0,"allocated":356352,"data_stored":136476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542175,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":950272,"data_stored":726844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705702,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":950272,"data_stored":726844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T03:40:06.775 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-10T03:40:06.775 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-10T03:40:06.775 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-10T03:40:06.776 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph health --format=json 2024-09-10T03:40:07.021 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:07.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:06 smithi169 ceph-mon[35668]: from='client.14528 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T03:40:07.289 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:06 smithi112 ceph-mon[27737]: from='client.14528 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T03:40:07.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.512+0000 7fb2a496a640 1 -- 172.21.15.112:0/1762904222 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2a0107930 msgr2=0x7fb2a0107d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:07.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.512+0000 7fb2a496a640 1 --2- 172.21.15.112:0/1762904222 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2a0107930 0x7fb2a0107d10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb298009c20 tx=0x7fb29802a970 comp rx=0 tx=0).stop 2024-09-10T03:40:07.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.512+0000 7fb2a496a640 1 -- 172.21.15.112:0/1762904222 shutdown_connections 2024-09-10T03:40:07.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.512+0000 7fb2a496a640 1 --2- 172.21.15.112:0/1762904222 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2a0101930 0x7fb2a0101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:07.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.512+0000 7fb2a496a640 1 --2- 172.21.15.112:0/1762904222 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2a0107930 0x7fb2a0107d10 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:07.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.512+0000 7fb2a496a640 1 -- 172.21.15.112:0/1762904222 >> 172.21.15.112:0/1762904222 conn(0x7fb2a00fd630 msgr2=0x7fb2a00ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:07.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.512+0000 7fb2a496a640 1 -- 172.21.15.112:0/1762904222 shutdown_connections 2024-09-10T03:40:07.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.512+0000 7fb2a496a640 1 -- 172.21.15.112:0/1762904222 wait complete. 2024-09-10T03:40:07.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.513+0000 7fb2a496a640 1 Processor -- start 2024-09-10T03:40:07.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.513+0000 7fb2a496a640 1 -- start start 2024-09-10T03:40:07.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.513+0000 7fb2a496a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2a0101930 0x7fb2a019d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:07.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.513+0000 7fb2a496a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2a0107930 0x7fb2a019dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:07.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.513+0000 7fb2a496a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a010ee00 con 0x7fb2a0107930 2024-09-10T03:40:07.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.513+0000 7fb2a496a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a010ec80 con 0x7fb2a0101930 2024-09-10T03:40:07.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.513+0000 7fb29dd9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2a0107930 0x7fb2a019dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:07.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.513+0000 7fb29dd9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2a0107930 0x7fb2a019dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:47966/0 (socket says 172.21.15.112:47966) 2024-09-10T03:40:07.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.513+0000 7fb29dd9b640 1 -- 172.21.15.112:0/2902615878 learned_addr learned my addr 172.21.15.112:0/2902615878 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:07.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.513+0000 7fb29e59c640 1 --2- 172.21.15.112:0/2902615878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2a0101930 0x7fb2a019d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:07.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.514+0000 7fb29dd9b640 1 -- 172.21.15.112:0/2902615878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2a0101930 msgr2=0x7fb2a019d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:07.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.514+0000 7fb29dd9b640 1 --2- 172.21.15.112:0/2902615878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2a0101930 0x7fb2a019d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:07.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.514+0000 7fb29dd9b640 1 -- 172.21.15.112:0/2902615878 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a01977f0 con 0x7fb2a0107930 2024-09-10T03:40:07.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.514+0000 7fb29dd9b640 1 --2- 172.21.15.112:0/2902615878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2a0107930 0x7fb2a019dbb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb28800d6f0 tx=0x7fb28800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:07.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.514+0000 7fb2977fe640 1 -- 172.21.15.112:0/2902615878 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb288017070 con 0x7fb2a0107930 2024-09-10T03:40:07.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.514+0000 7fb2977fe640 1 -- 172.21.15.112:0/2902615878 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb28800de00 con 0x7fb2a0107930 2024-09-10T03:40:07.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.514+0000 7fb2977fe640 1 -- 172.21.15.112:0/2902615878 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb288004bb0 con 0x7fb2a0107930 2024-09-10T03:40:07.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.514+0000 7fb2a496a640 1 -- 172.21.15.112:0/2902615878 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2a0197ae0 con 0x7fb2a0107930 2024-09-10T03:40:07.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.514+0000 7fb2a496a640 1 -- 172.21.15.112:0/2902615878 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2a0198070 con 0x7fb2a0107930 2024-09-10T03:40:07.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.515+0000 7fb2a496a640 1 -- 172.21.15.112:0/2902615878 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2a0103070 con 0x7fb2a0107930 2024-09-10T03:40:07.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.515+0000 7fb2977fe640 1 -- 172.21.15.112:0/2902615878 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fb288009a30 con 0x7fb2a0107930 2024-09-10T03:40:07.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.516+0000 7fb2977fe640 1 --2- 172.21.15.112:0/2902615878 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb28c06de90 0x7fb28c070350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:07.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.516+0000 7fb29e59c640 1 --2- 172.21.15.112:0/2902615878 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb28c06de90 0x7fb28c070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:07.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.516+0000 7fb2977fe640 1 -- 172.21.15.112:0/2902615878 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7fb28808d2f0 con 0x7fb2a0107930 2024-09-10T03:40:07.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.517+0000 7fb29e59c640 1 --2- 172.21.15.112:0/2902615878 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb28c06de90 0x7fb28c070350 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb298009430 tx=0x7fb29803c040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:07.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.521+0000 7fb2977fe640 1 -- 172.21.15.112:0/2902615878 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb28805ef00 con 0x7fb2a0107930 2024-09-10T03:40:07.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.724+0000 7fb2a496a640 1 -- 172.21.15.112:0/2902615878 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb2a0109ea0 con 0x7fb2a0107930 2024-09-10T03:40:07.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.725+0000 7fb2977fe640 1 -- 172.21.15.112:0/2902615878 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fb28808d110 con 0x7fb2a0107930 2024-09-10T03:40:07.726 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:40:07.726 INFO:teuthology.orchestra.run.smithi112.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-10T03:40:07.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.727+0000 7fb2a496a640 1 -- 172.21.15.112:0/2902615878 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb28c06de90 msgr2=0x7fb28c070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:07.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.727+0000 7fb2a496a640 1 --2- 172.21.15.112:0/2902615878 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb28c06de90 0x7fb28c070350 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb298009430 tx=0x7fb29803c040 comp rx=0 tx=0).stop 2024-09-10T03:40:07.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.727+0000 7fb2a496a640 1 -- 172.21.15.112:0/2902615878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2a0107930 msgr2=0x7fb2a019dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:07.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.727+0000 7fb2a496a640 1 --2- 172.21.15.112:0/2902615878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2a0107930 0x7fb2a019dbb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb28800d6f0 tx=0x7fb28800dbb0 comp rx=0 tx=0).stop 2024-09-10T03:40:07.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.727+0000 7fb2a496a640 1 -- 172.21.15.112:0/2902615878 shutdown_connections 2024-09-10T03:40:07.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.727+0000 7fb2a496a640 1 --2- 172.21.15.112:0/2902615878 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb28c06de90 0x7fb28c070350 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:07.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.727+0000 7fb2a496a640 1 --2- 172.21.15.112:0/2902615878 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2a0107930 0x7fb2a019dbb0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:07.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.727+0000 7fb2a496a640 1 --2- 172.21.15.112:0/2902615878 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb2a0101930 0x7fb2a019d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:07.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.727+0000 7fb2a496a640 1 -- 172.21.15.112:0/2902615878 >> 172.21.15.112:0/2902615878 conn(0x7fb2a00fd630 msgr2=0x7fb2a00fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:07.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.727+0000 7fb2a496a640 1 -- 172.21.15.112:0/2902615878 shutdown_connections 2024-09-10T03:40:07.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:07.727+0000 7fb2a496a640 1 -- 172.21.15.112:0/2902615878 wait complete. 2024-09-10T03:40:08.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:07 smithi112 ceph-mon[27737]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:08.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:07 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2902615878' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-10T03:40:08.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:07 smithi169 ceph-mon[35668]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:08.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:07 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/2902615878' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-10T03:40:08.321 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-10T03:40:08.322 INFO:tasks.cephadm:Setup complete, yielding 2024-09-10T03:40:08.322 INFO:teuthology.run_tasks:Running task print... 2024-09-10T03:40:08.330 INFO:teuthology.task.print:**** done end installing squid cephadm ... 2024-09-10T03:40:08.330 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-10T03:40:08.339 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi112.front.sepia.ceph.com 2024-09-10T03:40:08.339 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-09-10T03:40:08.587 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.036+0000 7f97c509e640 1 -- 172.21.15.112:0/4195544071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97c0101a80 msgr2=0x7f97c0101f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.036+0000 7f97c509e640 1 --2- 172.21.15.112:0/4195544071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97c0101a80 0x7f97c0101f00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009880 tx=0x7f97a802a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.036+0000 7f97c509e640 1 -- 172.21.15.112:0/4195544071 shutdown_connections 2024-09-10T03:40:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.036+0000 7f97c509e640 1 --2- 172.21.15.112:0/4195544071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97c0101a80 0x7f97c0101f00 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.036+0000 7f97c509e640 1 --2- 172.21.15.112:0/4195544071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97c0100880 0x7f97c0100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.036+0000 7f97c509e640 1 -- 172.21.15.112:0/4195544071 >> 172.21.15.112:0/4195544071 conn(0x7f97c00fc010 msgr2=0x7f97c00fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.036+0000 7f97c509e640 1 -- 172.21.15.112:0/4195544071 shutdown_connections 2024-09-10T03:40:09.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.036+0000 7f97c509e640 1 -- 172.21.15.112:0/4195544071 wait complete. 2024-09-10T03:40:09.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.036+0000 7f97c509e640 1 Processor -- start 2024-09-10T03:40:09.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.036+0000 7f97c509e640 1 -- start start 2024-09-10T03:40:09.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.037+0000 7f97c509e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97c0100880 0x7f97c006adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:09.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.037+0000 7f97c509e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97c0101a80 0x7f97c019c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:09.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.037+0000 7f97c509e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f97c01096c0 con 0x7f97c0101a80 2024-09-10T03:40:09.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.037+0000 7f97c509e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f97c0109540 con 0x7f97c0100880 2024-09-10T03:40:09.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.037+0000 7f97b7fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97c0101a80 0x7f97c019c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:09.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.037+0000 7f97b7fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97c0101a80 0x7f97c019c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:47998/0 (socket says 172.21.15.112:47998) 2024-09-10T03:40:09.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.037+0000 7f97b7fff640 1 -- 172.21.15.112:0/514875996 learned_addr learned my addr 172.21.15.112:0/514875996 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:09.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.037+0000 7f97bed9d640 1 --2- 172.21.15.112:0/514875996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97c0100880 0x7f97c006adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:09.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.037+0000 7f97b7fff640 1 -- 172.21.15.112:0/514875996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97c0100880 msgr2=0x7f97c006adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:09.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.037+0000 7f97b7fff640 1 --2- 172.21.15.112:0/514875996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97c0100880 0x7f97c006adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:09.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.037+0000 7f97b7fff640 1 -- 172.21.15.112:0/514875996 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c019c7c0 con 0x7f97c0101a80 2024-09-10T03:40:09.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.037+0000 7f97b7fff640 1 --2- 172.21.15.112:0/514875996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97c0101a80 0x7f97c019c200 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f97a8006010 tx=0x7f97a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:09.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.037+0000 7f97bcd99640 1 -- 172.21.15.112:0/514875996 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f97a803f070 con 0x7f97c0101a80 2024-09-10T03:40:09.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.037+0000 7f97bcd99640 1 -- 172.21.15.112:0/514875996 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f97a8004ac0 con 0x7f97c0101a80 2024-09-10T03:40:09.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.037+0000 7f97c509e640 1 -- 172.21.15.112:0/514875996 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c019ca50 con 0x7f97c0101a80 2024-09-10T03:40:09.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.037+0000 7f97bcd99640 1 -- 172.21.15.112:0/514875996 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f97a8035030 con 0x7f97c0101a80 2024-09-10T03:40:09.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.037+0000 7f97c509e640 1 -- 172.21.15.112:0/514875996 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c019cf80 con 0x7f97c0101a80 2024-09-10T03:40:09.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.038+0000 7f97c509e640 1 -- 172.21.15.112:0/514875996 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97c0100c80 con 0x7f97c0101a80 2024-09-10T03:40:09.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.039+0000 7f97bcd99640 1 -- 172.21.15.112:0/514875996 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f97a8042070 con 0x7f97c0101a80 2024-09-10T03:40:09.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.039+0000 7f97bcd99640 1 --2- 172.21.15.112:0/514875996 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f97ac06de30 0x7f97ac0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:09.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.039+0000 7f97bcd99640 1 -- 172.21.15.112:0/514875996 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f97a80ae450 con 0x7f97c0101a80 2024-09-10T03:40:09.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.039+0000 7f97bed9d640 1 --2- 172.21.15.112:0/514875996 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f97ac06de30 0x7f97ac0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:09.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.040+0000 7f97bed9d640 1 --2- 172.21.15.112:0/514875996 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f97ac06de30 0x7f97ac0702f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f97b0005fd0 tx=0x7f97b000b390 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:09.062 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.044+0000 7f97bcd99640 1 -- 172.21.15.112:0/514875996 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f97a807ff50 con 0x7f97c0101a80 2024-09-10T03:40:09.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.211+0000 7f97c509e640 1 -- 172.21.15.112:0/514875996 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) -- 0x7f97c0062fb0 con 0x7f97c0101a80 2024-09-10T03:40:09.216 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.215+0000 7f97bcd99640 1 -- 172.21.15.112:0/514875996 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v42)=0 v42) ==== 143+0+0 (secure 0 0 0) 0x7f97a80ae270 con 0x7f97c0101a80 2024-09-10T03:40:09.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.217+0000 7f97c509e640 1 -- 172.21.15.112:0/514875996 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f97ac06de30 msgr2=0x7f97ac0702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:09.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.217+0000 7f97c509e640 1 --2- 172.21.15.112:0/514875996 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f97ac06de30 0x7f97ac0702f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f97b0005fd0 tx=0x7f97b000b390 comp rx=0 tx=0).stop 2024-09-10T03:40:09.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.217+0000 7f97c509e640 1 -- 172.21.15.112:0/514875996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97c0101a80 msgr2=0x7f97c019c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:09.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.217+0000 7f97c509e640 1 --2- 172.21.15.112:0/514875996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97c0101a80 0x7f97c019c200 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f97a8006010 tx=0x7f97a80046c0 comp rx=0 tx=0).stop 2024-09-10T03:40:09.219 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.217+0000 7f97c509e640 1 -- 172.21.15.112:0/514875996 shutdown_connections 2024-09-10T03:40:09.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.217+0000 7f97c509e640 1 --2- 172.21.15.112:0/514875996 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f97ac06de30 0x7f97ac0702f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:09.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.217+0000 7f97c509e640 1 --2- 172.21.15.112:0/514875996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f97c0101a80 0x7f97c019c200 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:09.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.217+0000 7f97c509e640 1 --2- 172.21.15.112:0/514875996 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f97c0100880 0x7f97c006adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:09.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.217+0000 7f97c509e640 1 -- 172.21.15.112:0/514875996 >> 172.21.15.112:0/514875996 conn(0x7f97c00fc010 msgr2=0x7f97c0104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:09.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.217+0000 7f97c509e640 1 -- 172.21.15.112:0/514875996 shutdown_connections 2024-09-10T03:40:09.220 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:09.217+0000 7f97c509e640 1 -- 172.21.15.112:0/514875996 wait complete. 2024-09-10T03:40:09.834 INFO:teuthology.run_tasks:Running task print... 2024-09-10T03:40:09.843 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-09-10T03:40:09.844 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-10T03:40:09.851 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi112.front.sepia.ceph.com 2024-09-10T03:40:09.851 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- bash -c 'ceph orch status' 2024-09-10T03:40:10.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:09 smithi112 ceph-mon[27737]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:10.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:09 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/514875996' entity='client.admin' 2024-09-10T03:40:10.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:40:10.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:40:10.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:40:10.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:09 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:10.099 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:10.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:09 smithi169 ceph-mon[35668]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:10.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:09 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/514875996' entity='client.admin' 2024-09-10T03:40:10.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:40:10.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:40:10.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:40:10.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:09 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:10.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.547+0000 7fc61a129640 1 -- 172.21.15.112:0/611077653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc614102b90 msgr2=0x7fc614103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:10.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.547+0000 7fc61a129640 1 --2- 172.21.15.112:0/611077653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc614102b90 0x7fc614103010 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc608009880 tx=0x7fc60802a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:10.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.547+0000 7fc61a129640 1 -- 172.21.15.112:0/611077653 shutdown_connections 2024-09-10T03:40:10.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.547+0000 7fc61a129640 1 --2- 172.21.15.112:0/611077653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc614102b90 0x7fc614103010 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:10.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.547+0000 7fc61a129640 1 --2- 172.21.15.112:0/611077653 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc614101990 0x7fc614101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:10.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.547+0000 7fc61a129640 1 -- 172.21.15.112:0/611077653 >> 172.21.15.112:0/611077653 conn(0x7fc6140fd140 msgr2=0x7fc6140ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:10.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.547+0000 7fc61a129640 1 -- 172.21.15.112:0/611077653 shutdown_connections 2024-09-10T03:40:10.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.547+0000 7fc61a129640 1 -- 172.21.15.112:0/611077653 wait complete. 2024-09-10T03:40:10.595 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.548+0000 7fc61a129640 1 Processor -- start 2024-09-10T03:40:10.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.548+0000 7fc61a129640 1 -- start start 2024-09-10T03:40:10.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.548+0000 7fc61a129640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc614101990 0x7fc6141974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:10.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.548+0000 7fc61a129640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc614102b90 0x7fc6141979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:10.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.548+0000 7fc61a129640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc61410a7b0 con 0x7fc614101990 2024-09-10T03:40:10.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.548+0000 7fc61a129640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fc61410a630 con 0x7fc614102b90 2024-09-10T03:40:10.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.548+0000 7fc6137fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc614101990 0x7fc6141974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:10.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.548+0000 7fc6137fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc614101990 0x7fc6141974a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:48020/0 (socket says 172.21.15.112:48020) 2024-09-10T03:40:10.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.548+0000 7fc6137fe640 1 -- 172.21.15.112:0/269819276 learned_addr learned my addr 172.21.15.112:0/269819276 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:10.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.548+0000 7fc612ffd640 1 --2- 172.21.15.112:0/269819276 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc614102b90 0x7fc6141979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:10.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.549+0000 7fc6137fe640 1 -- 172.21.15.112:0/269819276 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc614102b90 msgr2=0x7fc6141979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:10.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.549+0000 7fc6137fe640 1 --2- 172.21.15.112:0/269819276 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc614102b90 0x7fc6141979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:10.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.549+0000 7fc6137fe640 1 -- 172.21.15.112:0/269819276 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc614197fd0 con 0x7fc614101990 2024-09-10T03:40:10.596 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.549+0000 7fc6137fe640 1 --2- 172.21.15.112:0/269819276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc614101990 0x7fc6141974a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc60400c8f0 tx=0x7fc60400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:10.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.549+0000 7fc610ff9640 1 -- 172.21.15.112:0/269819276 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fc604004020 con 0x7fc614101990 2024-09-10T03:40:10.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.549+0000 7fc610ff9640 1 -- 172.21.15.112:0/269819276 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc604007e10 con 0x7fc614101990 2024-09-10T03:40:10.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.549+0000 7fc610ff9640 1 -- 172.21.15.112:0/269819276 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fc604003e10 con 0x7fc614101990 2024-09-10T03:40:10.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.549+0000 7fc61a129640 1 -- 172.21.15.112:0/269819276 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61419ca40 con 0x7fc614101990 2024-09-10T03:40:10.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.549+0000 7fc61a129640 1 -- 172.21.15.112:0/269819276 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc61419cfa0 con 0x7fc614101990 2024-09-10T03:40:10.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.550+0000 7fc61a129640 1 -- 172.21.15.112:0/269819276 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc614101d90 con 0x7fc614101990 2024-09-10T03:40:10.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.550+0000 7fc610ff9640 1 -- 172.21.15.112:0/269819276 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fc604007950 con 0x7fc614101990 2024-09-10T03:40:10.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.551+0000 7fc610ff9640 1 --2- 172.21.15.112:0/269819276 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fc5fc06de30 0x7fc5fc0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:10.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.551+0000 7fc612ffd640 1 --2- 172.21.15.112:0/269819276 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fc5fc06de30 0x7fc5fc0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:10.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.551+0000 7fc610ff9640 1 -- 172.21.15.112:0/269819276 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7fc60408d9e0 con 0x7fc614101990 2024-09-10T03:40:10.597 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.551+0000 7fc612ffd640 1 --2- 172.21.15.112:0/269819276 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fc5fc06de30 0x7fc5fc0702f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc614198950 tx=0x7fc608005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:10.598 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.555+0000 7fc610ff9640 1 -- 172.21.15.112:0/269819276 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc604015330 con 0x7fc614101990 2024-09-10T03:40:10.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.735+0000 7fc61a129640 1 -- 172.21.15.112:0/269819276 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fc614107110 con 0x7fc5fc06de30 2024-09-10T03:40:10.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.736+0000 7fc610ff9640 1 -- 172.21.15.112:0/269819276 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fc614107110 con 0x7fc5fc06de30 2024-09-10T03:40:10.738 INFO:teuthology.orchestra.run.smithi112.stdout:Backend: cephadm 2024-09-10T03:40:10.738 INFO:teuthology.orchestra.run.smithi112.stdout:Available: Yes 2024-09-10T03:40:10.738 INFO:teuthology.orchestra.run.smithi112.stdout:Paused: No 2024-09-10T03:40:10.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.738+0000 7fc61a129640 1 -- 172.21.15.112:0/269819276 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fc5fc06de30 msgr2=0x7fc5fc0702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:10.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.738+0000 7fc61a129640 1 --2- 172.21.15.112:0/269819276 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fc5fc06de30 0x7fc5fc0702f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc614198950 tx=0x7fc608005a20 comp rx=0 tx=0).stop 2024-09-10T03:40:10.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.738+0000 7fc61a129640 1 -- 172.21.15.112:0/269819276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc614101990 msgr2=0x7fc6141974a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:10.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.738+0000 7fc61a129640 1 --2- 172.21.15.112:0/269819276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc614101990 0x7fc6141974a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc60400c8f0 tx=0x7fc60400cdb0 comp rx=0 tx=0).stop 2024-09-10T03:40:10.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.738+0000 7fc61a129640 1 -- 172.21.15.112:0/269819276 shutdown_connections 2024-09-10T03:40:10.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.738+0000 7fc61a129640 1 --2- 172.21.15.112:0/269819276 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fc5fc06de30 0x7fc5fc0702f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:10.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.738+0000 7fc61a129640 1 --2- 172.21.15.112:0/269819276 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc614102b90 0x7fc6141979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:10.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.738+0000 7fc61a129640 1 --2- 172.21.15.112:0/269819276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc614101990 0x7fc6141974a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:10.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.738+0000 7fc61a129640 1 -- 172.21.15.112:0/269819276 >> 172.21.15.112:0/269819276 conn(0x7fc6140fd140 msgr2=0x7fc6140ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:10.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.738+0000 7fc61a129640 1 -- 172.21.15.112:0/269819276 shutdown_connections 2024-09-10T03:40:10.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:10.738+0000 7fc61a129640 1 -- 172.21.15.112:0/269819276 wait complete. 2024-09-10T03:40:11.422 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- bash -c 'ceph orch ps' 2024-09-10T03:40:11.666 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:12.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.158+0000 7fcba888b640 1 -- 172.21.15.112:0/1271859715 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba4100880 msgr2=0x7fcba4100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:12.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.158+0000 7fcba888b640 1 --2- 172.21.15.112:0/1271859715 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba4100880 0x7fcba4100c80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcb94009880 tx=0x7fcb9402a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:12.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.158+0000 7fcba888b640 1 -- 172.21.15.112:0/1271859715 shutdown_connections 2024-09-10T03:40:12.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.158+0000 7fcba888b640 1 --2- 172.21.15.112:0/1271859715 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcba4101a80 0x7fcba4101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:12.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.158+0000 7fcba888b640 1 --2- 172.21.15.112:0/1271859715 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba4100880 0x7fcba4100c80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:12.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.158+0000 7fcba888b640 1 -- 172.21.15.112:0/1271859715 >> 172.21.15.112:0/1271859715 conn(0x7fcba40fc010 msgr2=0x7fcba40fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:12.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.158+0000 7fcba888b640 1 -- 172.21.15.112:0/1271859715 shutdown_connections 2024-09-10T03:40:12.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.158+0000 7fcba888b640 1 -- 172.21.15.112:0/1271859715 wait complete. 2024-09-10T03:40:12.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.159+0000 7fcba888b640 1 Processor -- start 2024-09-10T03:40:12.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.159+0000 7fcba888b640 1 -- start start 2024-09-10T03:40:12.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.159+0000 7fcba888b640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcba4100880 0x7fcba4196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:12.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.159+0000 7fcba888b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba4101a80 0x7fcba4196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:12.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.159+0000 7fcba888b640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcba4109720 con 0x7fcba4101a80 2024-09-10T03:40:12.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.159+0000 7fcba888b640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcba41095a0 con 0x7fcba4100880 2024-09-10T03:40:12.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.159+0000 7fcba1d9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba4101a80 0x7fcba4196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:12.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.159+0000 7fcba1d9b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba4101a80 0x7fcba4196950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:48032/0 (socket says 172.21.15.112:48032) 2024-09-10T03:40:12.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.159+0000 7fcba1d9b640 1 -- 172.21.15.112:0/442008408 learned_addr learned my addr 172.21.15.112:0/442008408 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:12.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.159+0000 7fcba259c640 1 --2- 172.21.15.112:0/442008408 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcba4100880 0x7fcba4196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:12.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.159+0000 7fcba1d9b640 1 -- 172.21.15.112:0/442008408 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcba4100880 msgr2=0x7fcba4196410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:12.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.159+0000 7fcba1d9b640 1 --2- 172.21.15.112:0/442008408 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcba4100880 0x7fcba4196410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:12.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.159+0000 7fcba1d9b640 1 -- 172.21.15.112:0/442008408 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba4196fd0 con 0x7fcba4101a80 2024-09-10T03:40:12.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.160+0000 7fcba1d9b640 1 --2- 172.21.15.112:0/442008408 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba4101a80 0x7fcba4196950 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800b420 tx=0x7fcb9800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:12.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.160+0000 7fcb937fe640 1 -- 172.21.15.112:0/442008408 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fcb98011630 con 0x7fcba4101a80 2024-09-10T03:40:12.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.160+0000 7fcba888b640 1 -- 172.21.15.112:0/442008408 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcba419ba10 con 0x7fcba4101a80 2024-09-10T03:40:12.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.160+0000 7fcb937fe640 1 -- 172.21.15.112:0/442008408 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcb98016070 con 0x7fcba4101a80 2024-09-10T03:40:12.174 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.160+0000 7fcb937fe640 1 -- 172.21.15.112:0/442008408 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fcb98010d40 con 0x7fcba4101a80 2024-09-10T03:40:12.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.160+0000 7fcba888b640 1 -- 172.21.15.112:0/442008408 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcba419c020 con 0x7fcba4101a80 2024-09-10T03:40:12.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.161+0000 7fcba888b640 1 -- 172.21.15.112:0/442008408 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcba4100c80 con 0x7fcba4101a80 2024-09-10T03:40:12.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.161+0000 7fcb937fe640 1 -- 172.21.15.112:0/442008408 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fcb9801b460 con 0x7fcba4101a80 2024-09-10T03:40:12.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.162+0000 7fcb937fe640 1 --2- 172.21.15.112:0/442008408 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fcb8c06de30 0x7fcb8c0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:12.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.162+0000 7fcba259c640 1 --2- 172.21.15.112:0/442008408 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fcb8c06de30 0x7fcb8c0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:12.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.162+0000 7fcb937fe640 1 -- 172.21.15.112:0/442008408 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7fcb9808d150 con 0x7fcba4101a80 2024-09-10T03:40:12.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.162+0000 7fcba259c640 1 --2- 172.21.15.112:0/442008408 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fcb8c06de30 0x7fcb8c0702f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcb94000c00 tx=0x7fcb94005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:12.175 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.166+0000 7fcb937fe640 1 -- 172.21.15.112:0/442008408 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcb9805b030 con 0x7fcba4101a80 2024-09-10T03:40:12.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:11 smithi169 ceph-mon[35668]: from='client.14540 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:40:12.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:11 smithi169 ceph-mon[35668]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:12.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:11 smithi112 ceph-mon[27737]: from='client.14540 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:40:12.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:11 smithi112 ceph-mon[27737]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:12.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.342+0000 7fcba888b640 1 -- 172.21.15.112:0/442008408 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fcba4106000 con 0x7fcb8c06de30 2024-09-10T03:40:12.352 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.350+0000 7fcb937fe640 1 -- 172.21.15.112:0/442008408 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3120 (secure 0 0 0) 0x7fcba4106000 con 0x7fcb8c06de30 2024-09-10T03:40:12.352 INFO:teuthology.orchestra.run.smithi112.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-10T03:40:12.352 INFO:teuthology.orchestra.run.smithi112.stdout:alertmanager.smithi112 smithi112 *:9093,9094 running (3m) 2m ago 5m 27.1M - 0.25.0 c8568f914cd2 28468ac02dbb 2024-09-10T03:40:12.352 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-exporter.smithi112 smithi112 running (5m) 2m ago 5m 7650k - 19.1.1-298-gb53b6d3e 54467abc55f8 0ff34f471a47 2024-09-10T03:40:12.352 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-exporter.smithi169 smithi169 running (4m) 41s ago 4m 5263k - 19.1.1-298-gb53b6d3e 54467abc55f8 f430c6826280 2024-09-10T03:40:12.352 INFO:teuthology.orchestra.run.smithi112.stdout:crash.smithi112 smithi112 running (5m) 2m ago 5m 6899k - 19.1.1-298-gb53b6d3e 54467abc55f8 f7d1df4e46c8 2024-09-10T03:40:12.353 INFO:teuthology.orchestra.run.smithi112.stdout:crash.smithi169 smithi169 running (4m) 41s ago 4m 6899k - 19.1.1-298-gb53b6d3e 54467abc55f8 a27310b2722f 2024-09-10T03:40:12.353 INFO:teuthology.orchestra.run.smithi112.stdout:grafana.smithi112 smithi112 *:3000 running (3m) 2m ago 4m 73.9M - 9.4.12 2bacad6d85d8 444b4b1d4b44 2024-09-10T03:40:12.353 INFO:teuthology.orchestra.run.smithi112.stdout:mgr.smithi112.pqrlrb smithi112 *:9283,8765,8443 running (6m) 2m ago 6m 489M - 19.1.1-298-gb53b6d3e 54467abc55f8 5bcbeef123a8 2024-09-10T03:40:12.353 INFO:teuthology.orchestra.run.smithi112.stdout:mgr.smithi169.bkwgjl smithi169 *:8443,9283,8765 running (4m) 41s ago 4m 440M - 19.1.1-298-gb53b6d3e 54467abc55f8 b78844ecd2af 2024-09-10T03:40:12.353 INFO:teuthology.orchestra.run.smithi112.stdout:mon.smithi112 smithi112 running (6m) 2m ago 6m 45.8M 2048M 19.1.1-298-gb53b6d3e 54467abc55f8 91597bdf09f7 2024-09-10T03:40:12.353 INFO:teuthology.orchestra.run.smithi112.stdout:mon.smithi169 smithi169 running (4m) 41s ago 4m 39.6M 2048M 19.1.1-298-gb53b6d3e 54467abc55f8 38476c275eb5 2024-09-10T03:40:12.353 INFO:teuthology.orchestra.run.smithi112.stdout:node-exporter.smithi112 smithi112 *:9100 running (5m) 2m ago 5m 15.8M - 1.5.0 0da6a335fe13 736b09a514a8 2024-09-10T03:40:12.353 INFO:teuthology.orchestra.run.smithi112.stdout:node-exporter.smithi169 smithi169 *:9100 running (4m) 41s ago 4m 14.8M - 1.5.0 0da6a335fe13 b6d2ee0493db 2024-09-10T03:40:12.353 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0 smithi112 running (3m) 2m ago 3m 49.1M 3946M 19.1.1-298-gb53b6d3e 54467abc55f8 274fcfdbe0d8 2024-09-10T03:40:12.353 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1 smithi112 running (2m) 2m ago 2m 48.2M 3946M 19.1.1-298-gb53b6d3e 54467abc55f8 c48f7eddf6a2 2024-09-10T03:40:12.353 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2 smithi112 running (2m) 2m ago 2m 15.0M 3946M 19.1.1-298-gb53b6d3e 54467abc55f8 2885646ffe21 2024-09-10T03:40:12.353 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3 smithi169 running (102s) 41s ago 101s 40.8M 4970M 19.1.1-298-gb53b6d3e 54467abc55f8 b55e9df427b0 2024-09-10T03:40:12.353 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4 smithi169 running (73s) 41s ago 73s 47.6M 4970M 19.1.1-298-gb53b6d3e 54467abc55f8 f3527032352f 2024-09-10T03:40:12.353 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5 smithi169 running (46s) 41s ago 46s 11.1M 4970M 19.1.1-298-gb53b6d3e 54467abc55f8 b4df34a99808 2024-09-10T03:40:12.354 INFO:teuthology.orchestra.run.smithi112.stdout:prometheus.smithi112 smithi112 *:9095 running (3m) 2m ago 4m 43.0M - 2.43.0 a07b618ecd1d 3cb9cb258756 2024-09-10T03:40:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.352+0000 7fcba888b640 1 -- 172.21.15.112:0/442008408 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fcb8c06de30 msgr2=0x7fcb8c0702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.352+0000 7fcba888b640 1 --2- 172.21.15.112:0/442008408 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fcb8c06de30 0x7fcb8c0702f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcb94000c00 tx=0x7fcb94005a20 comp rx=0 tx=0).stop 2024-09-10T03:40:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.352+0000 7fcba888b640 1 -- 172.21.15.112:0/442008408 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba4101a80 msgr2=0x7fcba4196950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.352+0000 7fcba888b640 1 --2- 172.21.15.112:0/442008408 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba4101a80 0x7fcba4196950 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800b420 tx=0x7fcb9800b8e0 comp rx=0 tx=0).stop 2024-09-10T03:40:12.354 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.352+0000 7fcba888b640 1 -- 172.21.15.112:0/442008408 shutdown_connections 2024-09-10T03:40:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.352+0000 7fcba888b640 1 --2- 172.21.15.112:0/442008408 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fcb8c06de30 0x7fcb8c0702f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.352+0000 7fcba888b640 1 --2- 172.21.15.112:0/442008408 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba4101a80 0x7fcba4196950 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.352+0000 7fcba888b640 1 --2- 172.21.15.112:0/442008408 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcba4100880 0x7fcba4196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.352+0000 7fcba888b640 1 -- 172.21.15.112:0/442008408 >> 172.21.15.112:0/442008408 conn(0x7fcba40fc010 msgr2=0x7fcba40fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.352+0000 7fcba888b640 1 -- 172.21.15.112:0/442008408 shutdown_connections 2024-09-10T03:40:12.355 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:12.353+0000 7fcba888b640 1 -- 172.21.15.112:0/442008408 wait complete. 2024-09-10T03:40:13.038 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- bash -c 'ceph orch ls' 2024-09-10T03:40:13.284 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:13.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.735+0000 7fea1b5f1640 1 -- 172.21.15.112:0/3126988268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea14100880 msgr2=0x7fea14100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:13.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.735+0000 7fea1b5f1640 1 --2- 172.21.15.112:0/3126988268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea14100880 0x7fea14100c80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fea10009880 tx=0x7fea1002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:13.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.735+0000 7fea1b5f1640 1 -- 172.21.15.112:0/3126988268 shutdown_connections 2024-09-10T03:40:13.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.735+0000 7fea1b5f1640 1 --2- 172.21.15.112:0/3126988268 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea14101a80 0x7fea14101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:13.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.735+0000 7fea1b5f1640 1 --2- 172.21.15.112:0/3126988268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea14100880 0x7fea14100c80 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:13.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.735+0000 7fea1b5f1640 1 -- 172.21.15.112:0/3126988268 >> 172.21.15.112:0/3126988268 conn(0x7fea140fc010 msgr2=0x7fea140fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:13.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.735+0000 7fea1b5f1640 1 -- 172.21.15.112:0/3126988268 shutdown_connections 2024-09-10T03:40:13.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.735+0000 7fea1b5f1640 1 -- 172.21.15.112:0/3126988268 wait complete. 2024-09-10T03:40:13.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.736+0000 7fea1b5f1640 1 Processor -- start 2024-09-10T03:40:13.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.736+0000 7fea1b5f1640 1 -- start start 2024-09-10T03:40:13.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.736+0000 7fea1b5f1640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea14100880 0x7fea141047a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:13.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.736+0000 7fea1b5f1640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea14101a80 0x7fea14102e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:13.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.736+0000 7fea1b5f1640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fea141096c0 con 0x7fea14100880 2024-09-10T03:40:13.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.736+0000 7fea1b5f1640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fea14109540 con 0x7fea14101a80 2024-09-10T03:40:13.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.736+0000 7fea1938d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea14100880 0x7fea141047a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:13.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.736+0000 7fea1938d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea14100880 0x7fea141047a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:48050/0 (socket says 172.21.15.112:48050) 2024-09-10T03:40:13.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.736+0000 7fea1938d640 1 -- 172.21.15.112:0/3168425174 learned_addr learned my addr 172.21.15.112:0/3168425174 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:13.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.736+0000 7fea18b8c640 1 --2- 172.21.15.112:0/3168425174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea14101a80 0x7fea14102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:13.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.736+0000 7fea1938d640 1 -- 172.21.15.112:0/3168425174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea14101a80 msgr2=0x7fea14102e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:13.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.736+0000 7fea1938d640 1 --2- 172.21.15.112:0/3168425174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea14101a80 0x7fea14102e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:13.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.736+0000 7fea1938d640 1 -- 172.21.15.112:0/3168425174 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea141033a0 con 0x7fea14100880 2024-09-10T03:40:13.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.736+0000 7fea1938d640 1 --2- 172.21.15.112:0/3168425174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea14100880 0x7fea141047a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fea10006010 tx=0x7fea100045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:13.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.737+0000 7fea0a7fc640 1 -- 172.21.15.112:0/3168425174 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fea1003f070 con 0x7fea14100880 2024-09-10T03:40:13.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.737+0000 7fea0a7fc640 1 -- 172.21.15.112:0/3168425174 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fea10004a50 con 0x7fea14100880 2024-09-10T03:40:13.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.737+0000 7fea1b5f1640 1 -- 172.21.15.112:0/3168425174 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea14103630 con 0x7fea14100880 2024-09-10T03:40:13.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.737+0000 7fea0a7fc640 1 -- 172.21.15.112:0/3168425174 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fea10035030 con 0x7fea14100880 2024-09-10T03:40:13.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.737+0000 7fea1b5f1640 1 -- 172.21.15.112:0/3168425174 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea14103b60 con 0x7fea14100880 2024-09-10T03:40:13.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.738+0000 7fea1b5f1640 1 -- 172.21.15.112:0/3168425174 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea14100c80 con 0x7fea14100880 2024-09-10T03:40:13.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.738+0000 7fea0a7fc640 1 -- 172.21.15.112:0/3168425174 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fea10042070 con 0x7fea14100880 2024-09-10T03:40:13.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.739+0000 7fea0a7fc640 1 --2- 172.21.15.112:0/3168425174 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fea0006de30 0x7fea000702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:13.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.739+0000 7fea18b8c640 1 --2- 172.21.15.112:0/3168425174 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fea0006de30 0x7fea000702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:13.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.739+0000 7fea0a7fc640 1 -- 172.21.15.112:0/3168425174 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7fea100ae400 con 0x7fea14100880 2024-09-10T03:40:13.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.739+0000 7fea18b8c640 1 --2- 172.21.15.112:0/3168425174 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fea0006de30 0x7fea000702f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fea04005fd0 tx=0x7fea04005850 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:13.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.744+0000 7fea0a7fc640 1 -- 172.21.15.112:0/3168425174 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fea10004c00 con 0x7fea14100880 2024-09-10T03:40:13.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.920+0000 7fea1b5f1640 1 -- 172.21.15.112:0/3168425174 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fea1406aa60 con 0x7fea0006de30 2024-09-10T03:40:13.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.925+0000 7fea0a7fc640 1 -- 172.21.15.112:0/3168425174 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+1310 (secure 0 0 0) 0x7fea1406aa60 con 0x7fea0006de30 2024-09-10T03:40:13.928 INFO:teuthology.orchestra.run.smithi112.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-10T03:40:13.928 INFO:teuthology.orchestra.run.smithi112.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2024-09-10T03:40:13.928 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-exporter 2/2 2m ago 5m * 2024-09-10T03:40:13.928 INFO:teuthology.orchestra.run.smithi112.stdout:crash 2/2 2m ago 5m * 2024-09-10T03:40:13.928 INFO:teuthology.orchestra.run.smithi112.stdout:grafana ?:3000 1/1 2m ago 5m count:1 2024-09-10T03:40:13.928 INFO:teuthology.orchestra.run.smithi112.stdout:mgr 2/2 2m ago 5m count:2 2024-09-10T03:40:13.928 INFO:teuthology.orchestra.run.smithi112.stdout:mon 2/2 2m ago 5m smithi112:172.21.15.112=smithi112;smithi169:172.21.15.169=smithi169;count:2 2024-09-10T03:40:13.928 INFO:teuthology.orchestra.run.smithi112.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2024-09-10T03:40:13.928 INFO:teuthology.orchestra.run.smithi112.stdout:osd 6 2m ago - 2024-09-10T03:40:13.928 INFO:teuthology.orchestra.run.smithi112.stdout:prometheus ?:9095 1/1 2m ago 5m count:1 2024-09-10T03:40:13.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.927+0000 7fea1b5f1640 1 -- 172.21.15.112:0/3168425174 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fea0006de30 msgr2=0x7fea000702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:13.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.927+0000 7fea1b5f1640 1 --2- 172.21.15.112:0/3168425174 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fea0006de30 0x7fea000702f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fea04005fd0 tx=0x7fea04005850 comp rx=0 tx=0).stop 2024-09-10T03:40:13.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.927+0000 7fea1b5f1640 1 -- 172.21.15.112:0/3168425174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea14100880 msgr2=0x7fea141047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:13.929 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.927+0000 7fea1b5f1640 1 --2- 172.21.15.112:0/3168425174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea14100880 0x7fea141047a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fea10006010 tx=0x7fea100045e0 comp rx=0 tx=0).stop 2024-09-10T03:40:13.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.927+0000 7fea1b5f1640 1 -- 172.21.15.112:0/3168425174 shutdown_connections 2024-09-10T03:40:13.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.928+0000 7fea1b5f1640 1 --2- 172.21.15.112:0/3168425174 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fea0006de30 0x7fea000702f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:13.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.928+0000 7fea1b5f1640 1 --2- 172.21.15.112:0/3168425174 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fea14101a80 0x7fea14102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:13.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.928+0000 7fea1b5f1640 1 --2- 172.21.15.112:0/3168425174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea14100880 0x7fea141047a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:13.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.928+0000 7fea1b5f1640 1 -- 172.21.15.112:0/3168425174 >> 172.21.15.112:0/3168425174 conn(0x7fea140fc010 msgr2=0x7fea1406a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:13.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.928+0000 7fea1b5f1640 1 -- 172.21.15.112:0/3168425174 shutdown_connections 2024-09-10T03:40:13.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:13.928+0000 7fea1b5f1640 1 -- 172.21.15.112:0/3168425174 wait complete. 2024-09-10T03:40:14.243 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:13 smithi112 ceph-mon[27737]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:40:14.243 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:13 smithi112 ceph-mon[27737]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:14.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:13 smithi169 ceph-mon[35668]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:40:14.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:13 smithi169 ceph-mon[35668]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:14.655 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- bash -c 'ceph orch host ls' 2024-09-10T03:40:14.903 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:15.264 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:14 smithi112 ceph-mon[27737]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:40:15.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:14 smithi169 ceph-mon[35668]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:40:15.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.346+0000 7f89fa599640 1 -- 172.21.15.112:0/2423518095 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f89f4100840 msgr2=0x7f89f4100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:15.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.347+0000 7f89fa599640 1 --2- 172.21.15.112:0/2423518095 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f89f4100840 0x7f89f4100c40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009880 tx=0x7f89e402a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:15.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.347+0000 7f89fa599640 1 -- 172.21.15.112:0/2423518095 shutdown_connections 2024-09-10T03:40:15.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.347+0000 7f89fa599640 1 --2- 172.21.15.112:0/2423518095 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89f4101a40 0x7f89f4101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:15.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.347+0000 7f89fa599640 1 --2- 172.21.15.112:0/2423518095 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f89f4100840 0x7f89f4100c40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:15.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.347+0000 7f89fa599640 1 -- 172.21.15.112:0/2423518095 >> 172.21.15.112:0/2423518095 conn(0x7f89f40fbff0 msgr2=0x7f89f40fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:15.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.347+0000 7f89fa599640 1 -- 172.21.15.112:0/2423518095 shutdown_connections 2024-09-10T03:40:15.356 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.347+0000 7f89fa599640 1 -- 172.21.15.112:0/2423518095 wait complete. 2024-09-10T03:40:15.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.347+0000 7f89fa599640 1 Processor -- start 2024-09-10T03:40:15.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.347+0000 7f89fa599640 1 -- start start 2024-09-10T03:40:15.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.348+0000 7f89fa599640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f89f4100840 0x7f89f41963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:15.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.348+0000 7f89fa599640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89f4101a40 0x7f89f4196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:15.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.348+0000 7f89fa599640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f89f4109680 con 0x7f89f4100840 2024-09-10T03:40:15.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.348+0000 7f89fa599640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f89f4109500 con 0x7f89f4101a40 2024-09-10T03:40:15.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.348+0000 7f89f3fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f89f4100840 0x7f89f41963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:15.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.348+0000 7f89f3fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f89f4100840 0x7f89f41963c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:60128/0 (socket says 172.21.15.112:60128) 2024-09-10T03:40:15.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.348+0000 7f89f3fff640 1 -- 172.21.15.112:0/1040104097 learned_addr learned my addr 172.21.15.112:0/1040104097 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:15.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.348+0000 7f89f37fe640 1 --2- 172.21.15.112:0/1040104097 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89f4101a40 0x7f89f4196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:15.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.348+0000 7f89f3fff640 1 -- 172.21.15.112:0/1040104097 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89f4101a40 msgr2=0x7f89f4196900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:15.357 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.348+0000 7f89f3fff640 1 --2- 172.21.15.112:0/1040104097 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89f4101a40 0x7f89f4196900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:15.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.348+0000 7f89f3fff640 1 -- 172.21.15.112:0/1040104097 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89f4196f80 con 0x7f89f4100840 2024-09-10T03:40:15.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.348+0000 7f89f3fff640 1 --2- 172.21.15.112:0/1040104097 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f89f4100840 0x7f89f41963c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f89e4006010 tx=0x7f89e40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:15.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.349+0000 7f89f17fa640 1 -- 172.21.15.112:0/1040104097 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f89e4035070 con 0x7f89f4100840 2024-09-10T03:40:15.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.349+0000 7f89fa599640 1 -- 172.21.15.112:0/1040104097 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89f4069b30 con 0x7f89f4100840 2024-09-10T03:40:15.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.349+0000 7f89f17fa640 1 -- 172.21.15.112:0/1040104097 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f89e40049d0 con 0x7f89f4100840 2024-09-10T03:40:15.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.349+0000 7f89fa599640 1 -- 172.21.15.112:0/1040104097 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89f406a0c0 con 0x7f89f4100840 2024-09-10T03:40:15.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.349+0000 7f89f17fa640 1 -- 172.21.15.112:0/1040104097 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f89e4031800 con 0x7f89f4100840 2024-09-10T03:40:15.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.350+0000 7f89fa599640 1 -- 172.21.15.112:0/1040104097 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89f4109e60 con 0x7f89f4100840 2024-09-10T03:40:15.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.350+0000 7f89f17fa640 1 -- 172.21.15.112:0/1040104097 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f89e4003cd0 con 0x7f89f4100840 2024-09-10T03:40:15.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.351+0000 7f89f17fa640 1 --2- 172.21.15.112:0/1040104097 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f89e006de30 0x7f89e00702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:15.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.351+0000 7f89f17fa640 1 -- 172.21.15.112:0/1040104097 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f89e403e070 con 0x7f89f4100840 2024-09-10T03:40:15.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.351+0000 7f89f37fe640 1 --2- 172.21.15.112:0/1040104097 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f89e006de30 0x7f89e00702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:15.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.351+0000 7f89f37fe640 1 --2- 172.21.15.112:0/1040104097 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f89e006de30 0x7f89e00702f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f89f4197870 tx=0x7f89dc00b390 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:15.359 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.359+0000 7f89f17fa640 1 -- 172.21.15.112:0/1040104097 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f89e407dbf0 con 0x7f89f4100840 2024-09-10T03:40:15.538 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.537+0000 7f89fa599640 1 -- 172.21.15.112:0/1040104097 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f89f4190470 con 0x7f89e006de30 2024-09-10T03:40:15.539 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.538+0000 7f89f17fa640 1 -- 172.21.15.112:0/1040104097 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+148 (secure 0 0 0) 0x7f89f4190470 con 0x7f89e006de30 2024-09-10T03:40:15.539 INFO:teuthology.orchestra.run.smithi112.stdout:HOST ADDR LABELS STATUS 2024-09-10T03:40:15.539 INFO:teuthology.orchestra.run.smithi112.stdout:smithi112 172.21.15.112 2024-09-10T03:40:15.539 INFO:teuthology.orchestra.run.smithi112.stdout:smithi169 172.21.15.169 2024-09-10T03:40:15.539 INFO:teuthology.orchestra.run.smithi112.stdout:2 hosts in cluster 2024-09-10T03:40:15.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.540+0000 7f89fa599640 1 -- 172.21.15.112:0/1040104097 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f89e006de30 msgr2=0x7f89e00702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:15.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.540+0000 7f89fa599640 1 --2- 172.21.15.112:0/1040104097 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f89e006de30 0x7f89e00702f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f89f4197870 tx=0x7f89dc00b390 comp rx=0 tx=0).stop 2024-09-10T03:40:15.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.540+0000 7f89fa599640 1 -- 172.21.15.112:0/1040104097 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f89f4100840 msgr2=0x7f89f41963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:15.541 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.540+0000 7f89fa599640 1 --2- 172.21.15.112:0/1040104097 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f89f4100840 0x7f89f41963c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f89e4006010 tx=0x7f89e40045e0 comp rx=0 tx=0).stop 2024-09-10T03:40:15.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.540+0000 7f89fa599640 1 -- 172.21.15.112:0/1040104097 shutdown_connections 2024-09-10T03:40:15.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.540+0000 7f89fa599640 1 --2- 172.21.15.112:0/1040104097 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f89e006de30 0x7f89e00702f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:15.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.540+0000 7f89fa599640 1 --2- 172.21.15.112:0/1040104097 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f89f4101a40 0x7f89f4196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:15.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.540+0000 7f89fa599640 1 --2- 172.21.15.112:0/1040104097 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f89f4100840 0x7f89f41963c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:15.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.540+0000 7f89fa599640 1 -- 172.21.15.112:0/1040104097 >> 172.21.15.112:0/1040104097 conn(0x7f89f40fbff0 msgr2=0x7f89f4104ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:15.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.540+0000 7f89fa599640 1 -- 172.21.15.112:0/1040104097 shutdown_connections 2024-09-10T03:40:15.542 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:15.540+0000 7f89fa599640 1 -- 172.21.15.112:0/1040104097 wait complete. 2024-09-10T03:40:16.171 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- bash -c 'ceph orch device ls' 2024-09-10T03:40:16.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:15 smithi169 ceph-mon[35668]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:16.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:15 smithi112 ceph-mon[27737]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:16.428 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.847+0000 7fd5eb8ee640 1 -- 172.21.15.112:0/2401081685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e4100840 msgr2=0x7fd5e4100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.847+0000 7fd5eb8ee640 1 --2- 172.21.15.112:0/2401081685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e4100840 0x7fd5e4100c40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0009880 tx=0x7fd5e002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.847+0000 7fd5eb8ee640 1 -- 172.21.15.112:0/2401081685 shutdown_connections 2024-09-10T03:40:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.847+0000 7fd5eb8ee640 1 --2- 172.21.15.112:0/2401081685 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5e4101a40 0x7fd5e4101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.847+0000 7fd5eb8ee640 1 --2- 172.21.15.112:0/2401081685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e4100840 0x7fd5e4100c40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.847+0000 7fd5eb8ee640 1 -- 172.21.15.112:0/2401081685 >> 172.21.15.112:0/2401081685 conn(0x7fd5e40fbff0 msgr2=0x7fd5e40fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.847+0000 7fd5eb8ee640 1 -- 172.21.15.112:0/2401081685 shutdown_connections 2024-09-10T03:40:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.847+0000 7fd5eb8ee640 1 -- 172.21.15.112:0/2401081685 wait complete. 2024-09-10T03:40:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.848+0000 7fd5eb8ee640 1 Processor -- start 2024-09-10T03:40:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.848+0000 7fd5eb8ee640 1 -- start start 2024-09-10T03:40:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.848+0000 7fd5eb8ee640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5e4100840 0x7fd5e4196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.848+0000 7fd5eb8ee640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e4101a40 0x7fd5e41968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.848+0000 7fd5eb8ee640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e4109660 con 0x7fd5e4101a40 2024-09-10T03:40:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.848+0000 7fd5eb8ee640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e41094e0 con 0x7fd5e4100840 2024-09-10T03:40:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.848+0000 7fd5e8e89640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e4101a40 0x7fd5e41968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.848+0000 7fd5e8e89640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e4101a40 0x7fd5e41968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:60144/0 (socket says 172.21.15.112:60144) 2024-09-10T03:40:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.848+0000 7fd5e8e89640 1 -- 172.21.15.112:0/3650309111 learned_addr learned my addr 172.21.15.112:0/3650309111 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.848+0000 7fd5e8e89640 1 -- 172.21.15.112:0/3650309111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5e4100840 msgr2=0x7fd5e4196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.848+0000 7fd5e8e89640 1 --2- 172.21.15.112:0/3650309111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5e4100840 0x7fd5e4196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.848+0000 7fd5e8e89640 1 -- 172.21.15.112:0/3650309111 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e4196ec0 con 0x7fd5e4101a40 2024-09-10T03:40:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.848+0000 7fd5e8e89640 1 --2- 172.21.15.112:0/3650309111 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e4101a40 0x7fd5e41968d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd5d400c8a0 tx=0x7fd5d400cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.849+0000 7fd5da7fc640 1 -- 172.21.15.112:0/3650309111 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fd5d40079b0 con 0x7fd5e4101a40 2024-09-10T03:40:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.849+0000 7fd5eb8ee640 1 -- 172.21.15.112:0/3650309111 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5e419b930 con 0x7fd5e4101a40 2024-09-10T03:40:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.849+0000 7fd5da7fc640 1 -- 172.21.15.112:0/3650309111 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd5d4007b50 con 0x7fd5e4101a40 2024-09-10T03:40:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.849+0000 7fd5da7fc640 1 -- 172.21.15.112:0/3650309111 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fd5d4018700 con 0x7fd5e4101a40 2024-09-10T03:40:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.849+0000 7fd5eb8ee640 1 -- 172.21.15.112:0/3650309111 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5e419be90 con 0x7fd5e4101a40 2024-09-10T03:40:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.850+0000 7fd5eb8ee640 1 -- 172.21.15.112:0/3650309111 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5e4100c40 con 0x7fd5e4101a40 2024-09-10T03:40:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.850+0000 7fd5da7fc640 1 -- 172.21.15.112:0/3650309111 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fd5d400f450 con 0x7fd5e4101a40 2024-09-10T03:40:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.850+0000 7fd5da7fc640 1 --2- 172.21.15.112:0/3650309111 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fd5d006de30 0x7fd5d00702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.851+0000 7fd5da7fc640 1 -- 172.21.15.112:0/3650309111 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7fd5d408db30 con 0x7fd5e4101a40 2024-09-10T03:40:16.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.851+0000 7fd5e968a640 1 --2- 172.21.15.112:0/3650309111 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fd5d006de30 0x7fd5d00702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:16.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.851+0000 7fd5e968a640 1 --2- 172.21.15.112:0/3650309111 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fd5d006de30 0x7fd5d00702f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd5e00096c0 tx=0x7fd5e0000bc0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:16.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:16.859+0000 7fd5da7fc640 1 -- 172.21.15.112:0/3650309111 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd5d4014030 con 0x7fd5e4101a40 2024-09-10T03:40:17.037 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:17.036+0000 7fd5eb8ee640 1 -- 172.21.15.112:0/3650309111 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fd5e4062630 con 0x7fd5d006de30 2024-09-10T03:40:17.038 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:17.037+0000 7fd5da7fc640 1 -- 172.21.15.112:0/3650309111 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+399 (secure 0 0 0) 0x7fd5e4062630 con 0x7fd5d006de30 2024-09-10T03:40:17.038 INFO:teuthology.orchestra.run.smithi112.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-09-10T03:40:17.038 INFO:teuthology.orchestra.run.smithi112.stdout:smithi112 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NV400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-09-10T03:40:17.038 INFO:teuthology.orchestra.run.smithi112.stdout:smithi169 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300LN400BGN 372G No 43s ago Has a FileSystem, LVM detected 2024-09-10T03:40:17.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:17.039+0000 7fd5eb8ee640 1 -- 172.21.15.112:0/3650309111 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fd5d006de30 msgr2=0x7fd5d00702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:17.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:17.039+0000 7fd5eb8ee640 1 --2- 172.21.15.112:0/3650309111 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fd5d006de30 0x7fd5d00702f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd5e00096c0 tx=0x7fd5e0000bc0 comp rx=0 tx=0).stop 2024-09-10T03:40:17.040 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:17.039+0000 7fd5eb8ee640 1 -- 172.21.15.112:0/3650309111 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e4101a40 msgr2=0x7fd5e41968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:17.041 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:17.039+0000 7fd5eb8ee640 1 --2- 172.21.15.112:0/3650309111 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e4101a40 0x7fd5e41968d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd5d400c8a0 tx=0x7fd5d400cd60 comp rx=0 tx=0).stop 2024-09-10T03:40:17.041 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:17.039+0000 7fd5eb8ee640 1 -- 172.21.15.112:0/3650309111 shutdown_connections 2024-09-10T03:40:17.041 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:17.039+0000 7fd5eb8ee640 1 --2- 172.21.15.112:0/3650309111 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fd5d006de30 0x7fd5d00702f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:17.041 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:17.039+0000 7fd5eb8ee640 1 --2- 172.21.15.112:0/3650309111 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e4101a40 0x7fd5e41968d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:17.041 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:17.039+0000 7fd5eb8ee640 1 --2- 172.21.15.112:0/3650309111 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd5e4100840 0x7fd5e4196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:17.041 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:17.039+0000 7fd5eb8ee640 1 -- 172.21.15.112:0/3650309111 >> 172.21.15.112:0/3650309111 conn(0x7fd5e40fbff0 msgr2=0x7fd5e4104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:17.041 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:17.040+0000 7fd5eb8ee640 1 -- 172.21.15.112:0/3650309111 shutdown_connections 2024-09-10T03:40:17.041 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:17.040+0000 7fd5eb8ee640 1 -- 172.21.15.112:0/3650309111 wait complete. 2024-09-10T03:40:17.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:16 smithi169 ceph-mon[35668]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:40:17.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:16 smithi112 ceph-mon[27737]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:40:17.656 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-10T03:40:17.666 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi112.front.sepia.ceph.com 2024-09-10T03:40:17.666 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-09-10T03:40:17.917 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:18.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:18 smithi169 ceph-mon[35668]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:18.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:18 smithi169 ceph-mon[35668]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:40:18.321 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:18 smithi112 ceph-mon[27737]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:18.321 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:18 smithi112 ceph-mon[27737]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:40:18.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.405+0000 7f4486e11640 1 -- 172.21.15.112:0/3263936287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4480101a40 msgr2=0x7f4480101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:18.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.405+0000 7f4486e11640 1 --2- 172.21.15.112:0/3263936287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4480101a40 0x7f4480101ec0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4468009c20 tx=0x7f446802a970 comp rx=0 tx=0).stop 2024-09-10T03:40:18.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.405+0000 7f4486e11640 1 -- 172.21.15.112:0/3263936287 shutdown_connections 2024-09-10T03:40:18.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.405+0000 7f4486e11640 1 --2- 172.21.15.112:0/3263936287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4480101a40 0x7f4480101ec0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:18.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.405+0000 7f4486e11640 1 --2- 172.21.15.112:0/3263936287 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4480100840 0x7f4480100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:18.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.405+0000 7f4486e11640 1 -- 172.21.15.112:0/3263936287 >> 172.21.15.112:0/3263936287 conn(0x7f44800fbff0 msgr2=0x7f44800fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:18.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.405+0000 7f4486e11640 1 -- 172.21.15.112:0/3263936287 shutdown_connections 2024-09-10T03:40:18.430 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.405+0000 7f4486e11640 1 -- 172.21.15.112:0/3263936287 wait complete. 2024-09-10T03:40:18.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.405+0000 7f4486e11640 1 Processor -- start 2024-09-10T03:40:18.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.405+0000 7f4486e11640 1 -- start start 2024-09-10T03:40:18.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f4486e11640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4480100840 0x7f44801975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:18.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f4486e11640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4480101a40 0x7f4480197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:18.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f4486e11640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4480109700 con 0x7f4480100840 2024-09-10T03:40:18.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f4486e11640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f4480109580 con 0x7f4480101a40 2024-09-10T03:40:18.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f4484bad640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4480100840 0x7f44801975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:18.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f4484bad640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4480100840 0x7f44801975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:60152/0 (socket says 172.21.15.112:60152) 2024-09-10T03:40:18.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f4484bad640 1 -- 172.21.15.112:0/2639393787 learned_addr learned my addr 172.21.15.112:0/2639393787 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:18.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f447ffff640 1 --2- 172.21.15.112:0/2639393787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4480101a40 0x7f4480197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:18.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f4484bad640 1 -- 172.21.15.112:0/2639393787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4480101a40 msgr2=0x7f4480197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:18.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f4484bad640 1 --2- 172.21.15.112:0/2639393787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4480101a40 0x7f4480197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:18.431 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f4484bad640 1 -- 172.21.15.112:0/2639393787 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4480198190 con 0x7f4480100840 2024-09-10T03:40:18.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f447ffff640 1 --2- 172.21.15.112:0/2639393787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4480101a40 0x7f4480197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T03:40:18.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f4484bad640 1 --2- 172.21.15.112:0/2639393787 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4480100840 0x7f44801975d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f447800b3d0 tx=0x7f447800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:18.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f447dffb640 1 -- 172.21.15.112:0/2639393787 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f4478015020 con 0x7f4480100840 2024-09-10T03:40:18.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f447dffb640 1 -- 172.21.15.112:0/2639393787 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4478011af0 con 0x7f4480100840 2024-09-10T03:40:18.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f4486e11640 1 -- 172.21.15.112:0/2639393787 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f448019cbd0 con 0x7f4480100840 2024-09-10T03:40:18.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f447dffb640 1 -- 172.21.15.112:0/2639393787 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f4478010da0 con 0x7f4480100840 2024-09-10T03:40:18.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.406+0000 7f4486e11640 1 -- 172.21.15.112:0/2639393787 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f448019d160 con 0x7f4480100840 2024-09-10T03:40:18.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.407+0000 7f4486e11640 1 -- 172.21.15.112:0/2639393787 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4480100cc0 con 0x7f4480100840 2024-09-10T03:40:18.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.408+0000 7f447dffb640 1 -- 172.21.15.112:0/2639393787 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f4478018020 con 0x7f4480100840 2024-09-10T03:40:18.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.408+0000 7f447dffb640 1 --2- 172.21.15.112:0/2639393787 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f446c06de90 0x7f446c070350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:18.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.408+0000 7f447dffb640 1 -- 172.21.15.112:0/2639393787 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f447808d360 con 0x7f4480100840 2024-09-10T03:40:18.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.408+0000 7f447ffff640 1 --2- 172.21.15.112:0/2639393787 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f446c06de90 0x7f446c070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:18.432 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.409+0000 7f447ffff640 1 --2- 172.21.15.112:0/2639393787 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f446c06de90 0x7f446c070350 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4468006010 tx=0x7f44680093f0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:18.433 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.413+0000 7f447dffb640 1 -- 172.21.15.112:0/2639393787 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f447801f020 con 0x7f4480100840 2024-09-10T03:40:18.603 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:18.602+0000 7f4486e11640 1 -- 172.21.15.112:0/2639393787 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) -- 0x7f44800fde90 con 0x7f446c06de90 2024-09-10T03:40:19.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:19 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-09-10T03:40:19.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:19 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-09-10T03:40:20.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:20.048+0000 7f447dffb640 1 -- 172.21.15.112:0/2639393787 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f44800fde90 con 0x7f446c06de90 2024-09-10T03:40:20.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:20.051+0000 7f4486e11640 1 -- 172.21.15.112:0/2639393787 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f446c06de90 msgr2=0x7f446c070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:20.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:20.051+0000 7f4486e11640 1 --2- 172.21.15.112:0/2639393787 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f446c06de90 0x7f446c070350 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4468006010 tx=0x7f44680093f0 comp rx=0 tx=0).stop 2024-09-10T03:40:20.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:20.051+0000 7f4486e11640 1 -- 172.21.15.112:0/2639393787 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4480100840 msgr2=0x7f44801975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:20.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:20.051+0000 7f4486e11640 1 --2- 172.21.15.112:0/2639393787 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4480100840 0x7f44801975d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f447800b3d0 tx=0x7f447800b890 comp rx=0 tx=0).stop 2024-09-10T03:40:20.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:20.051+0000 7f4486e11640 1 -- 172.21.15.112:0/2639393787 shutdown_connections 2024-09-10T03:40:20.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:20.051+0000 7f4486e11640 1 --2- 172.21.15.112:0/2639393787 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f446c06de90 0x7f446c070350 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:20.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:20.051+0000 7f4486e11640 1 --2- 172.21.15.112:0/2639393787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4480101a40 0x7f4480197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:20.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:20.051+0000 7f4486e11640 1 --2- 172.21.15.112:0/2639393787 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4480100840 0x7f44801975d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:20.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:20.051+0000 7f4486e11640 1 -- 172.21.15.112:0/2639393787 >> 172.21.15.112:0/2639393787 conn(0x7f44800fbff0 msgr2=0x7f44800fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:20.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:20.051+0000 7f4486e11640 1 -- 172.21.15.112:0/2639393787 shutdown_connections 2024-09-10T03:40:20.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:20.051+0000 7f4486e11640 1 -- 172.21.15.112:0/2639393787 wait complete. 2024-09-10T03:40:20.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:20 smithi169 ceph-mon[35668]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:40:20.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:20 smithi169 ceph-mon[35668]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:20.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:20 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-09-10T03:40:20.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:20 smithi169 ceph-mon[35668]: osdmap e40: 6 total, 6 up, 6 in 2024-09-10T03:40:20.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:20 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-09-10T03:40:20.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:20 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:40:20.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:20 smithi112 ceph-mon[27737]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:40:20.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:20 smithi112 ceph-mon[27737]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:20.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:20 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-09-10T03:40:20.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:20 smithi112 ceph-mon[27737]: osdmap e40: 6 total, 6 up, 6 in 2024-09-10T03:40:20.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:20 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-09-10T03:40:20.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:20 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:40:20.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:40:20.025+0000 7fd3aff0d640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-10T03:40:20.698 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- bash -c 'ceph fs dump' 2024-09-10T03:40:21.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:21 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-09-10T03:40:21.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:21 smithi169 ceph-mon[35668]: osdmap e41: 6 total, 6 up, 6 in 2024-09-10T03:40:21.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:21 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-09-10T03:40:21.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:21 smithi169 ceph-mon[35668]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-10T03:40:21.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:21 smithi169 ceph-mon[35668]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-10T03:40:21.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:21 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-09-10T03:40:21.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:21 smithi169 ceph-mon[35668]: osdmap e42: 6 total, 6 up, 6 in 2024-09-10T03:40:21.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:21 smithi169 ceph-mon[35668]: fsmap cephfs:0 2024-09-10T03:40:21.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:21 smithi169 ceph-mon[35668]: Saving service mds.cephfs spec with placement count:4 2024-09-10T03:40:21.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:21 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:21.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:21 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:40:21.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:21 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:40:21.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:21 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:40:21.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:21 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:21.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:21 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi112.ifnhnl", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-10T03:40:21.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:21 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi112.ifnhnl", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-10T03:40:21.278 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:21 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:40:21.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:21 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-09-10T03:40:21.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:21 smithi112 ceph-mon[27737]: osdmap e41: 6 total, 6 up, 6 in 2024-09-10T03:40:21.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:21 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-09-10T03:40:21.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:21 smithi112 ceph-mon[27737]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-10T03:40:21.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:21 smithi112 ceph-mon[27737]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-10T03:40:21.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:21 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-09-10T03:40:21.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:21 smithi112 ceph-mon[27737]: osdmap e42: 6 total, 6 up, 6 in 2024-09-10T03:40:21.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:21 smithi112 ceph-mon[27737]: fsmap cephfs:0 2024-09-10T03:40:21.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:21 smithi112 ceph-mon[27737]: Saving service mds.cephfs spec with placement count:4 2024-09-10T03:40:21.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:21 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:21.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:21 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:40:21.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:21 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:40:21.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:21 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:40:21.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:21 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:21.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:21 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi112.ifnhnl", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-10T03:40:21.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:21 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi112.ifnhnl", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-10T03:40:21.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:21 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:40:21.839 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:22.173 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:22 smithi112 ceph-mon[27737]: Deploying daemon mds.cephfs.smithi112.ifnhnl on smithi112 2024-09-10T03:40:22.173 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:22 smithi112 ceph-mon[27737]: pgmap v173: 65 pgs: 1 creating+activating, 19 creating+peering, 20 active+clean, 25 unknown; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:22.173 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:22 smithi112 ceph-mon[27737]: osdmap e43: 6 total, 6 up, 6 in 2024-09-10T03:40:22.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.404+0000 7f8473d6e640 1 -- 172.21.15.112:0/4264803493 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f846c0ffc50 msgr2=0x7f846c100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:22.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.404+0000 7f8473d6e640 1 --2- 172.21.15.112:0/4264803493 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f846c0ffc50 0x7f846c100050 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8460009880 tx=0x7f846002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:22.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.404+0000 7f8473d6e640 1 -- 172.21.15.112:0/4264803493 shutdown_connections 2024-09-10T03:40:22.405 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.404+0000 7f8473d6e640 1 --2- 172.21.15.112:0/4264803493 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f846c102c30 0x7f846c1030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:22.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.404+0000 7f8473d6e640 1 --2- 172.21.15.112:0/4264803493 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f846c0ffc50 0x7f846c100050 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:22.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.404+0000 7f8473d6e640 1 -- 172.21.15.112:0/4264803493 >> 172.21.15.112:0/4264803493 conn(0x7f846c0fb7a0 msgr2=0x7f846c0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:22.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.404+0000 7f8473d6e640 1 -- 172.21.15.112:0/4264803493 shutdown_connections 2024-09-10T03:40:22.406 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.404+0000 7f8473d6e640 1 -- 172.21.15.112:0/4264803493 wait complete. 2024-09-10T03:40:22.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.405+0000 7f8473d6e640 1 Processor -- start 2024-09-10T03:40:22.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.405+0000 7f8473d6e640 1 -- start start 2024-09-10T03:40:22.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.405+0000 7f8473d6e640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f846c0ffc50 0x7f846c197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:22.407 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.405+0000 7f8473d6e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f846c102c30 0x7f846c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:22.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.405+0000 7f8473d6e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f846c10a830 con 0x7f846c102c30 2024-09-10T03:40:22.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.405+0000 7f8473d6e640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f846c10a6b0 con 0x7f846c0ffc50 2024-09-10T03:40:22.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.405+0000 7f8471309640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f846c102c30 0x7f846c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:22.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.405+0000 7f8471309640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f846c102c30 0x7f846c197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:60156/0 (socket says 172.21.15.112:60156) 2024-09-10T03:40:22.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.405+0000 7f8471309640 1 -- 172.21.15.112:0/2871363840 learned_addr learned my addr 172.21.15.112:0/2871363840 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:22.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.406+0000 7f8471b0a640 1 --2- 172.21.15.112:0/2871363840 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f846c0ffc50 0x7f846c197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:22.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.406+0000 7f8471309640 1 -- 172.21.15.112:0/2871363840 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f846c0ffc50 msgr2=0x7f846c197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:22.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.406+0000 7f8471309640 1 --2- 172.21.15.112:0/2871363840 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f846c0ffc50 0x7f846c197520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:22.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.406+0000 7f8471309640 1 -- 172.21.15.112:0/2871363840 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f846c1980e0 con 0x7f846c102c30 2024-09-10T03:40:22.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.406+0000 7f8471309640 1 --2- 172.21.15.112:0/2871363840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f846c102c30 0x7f846c197a60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f846800b420 tx=0x7f846800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:22.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.406+0000 7f845effd640 1 -- 172.21.15.112:0/2871363840 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8468011630 con 0x7f846c102c30 2024-09-10T03:40:22.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.406+0000 7f845effd640 1 -- 172.21.15.112:0/2871363840 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8468016070 con 0x7f846c102c30 2024-09-10T03:40:22.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.406+0000 7f8473d6e640 1 -- 172.21.15.112:0/2871363840 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f846c19cb20 con 0x7f846c102c30 2024-09-10T03:40:22.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.406+0000 7f845effd640 1 -- 172.21.15.112:0/2871363840 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8468010d40 con 0x7f846c102c30 2024-09-10T03:40:22.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.406+0000 7f8473d6e640 1 -- 172.21.15.112:0/2871363840 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f846c19d130 con 0x7f846c102c30 2024-09-10T03:40:22.409 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.407+0000 7f8473d6e640 1 -- 172.21.15.112:0/2871363840 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f846c100050 con 0x7f846c102c30 2024-09-10T03:40:22.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.408+0000 7f845effd640 1 -- 172.21.15.112:0/2871363840 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f846801b460 con 0x7f846c102c30 2024-09-10T03:40:22.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.408+0000 7f845effd640 1 --2- 172.21.15.112:0/2871363840 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f845806de30 0x7f84580702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:22.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.408+0000 7f845effd640 1 -- 172.21.15.112:0/2871363840 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5468+0+0 (secure 0 0 0) 0x7f846808d4d0 con 0x7f846c102c30 2024-09-10T03:40:22.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.408+0000 7f8471b0a640 1 --2- 172.21.15.112:0/2871363840 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f845806de30 0x7f84580702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:22.410 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.409+0000 7f8471b0a640 1 --2- 172.21.15.112:0/2871363840 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f845806de30 0x7f84580702f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8460000c00 tx=0x7f8460005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:22.414 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.413+0000 7f845effd640 1 -- 172.21.15.112:0/2871363840 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f846805b070 con 0x7f846c102c30 2024-09-10T03:40:22.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:22 smithi169 ceph-mon[35668]: Deploying daemon mds.cephfs.smithi112.ifnhnl on smithi112 2024-09-10T03:40:22.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:22 smithi169 ceph-mon[35668]: pgmap v173: 65 pgs: 1 creating+activating, 19 creating+peering, 20 active+clean, 25 unknown; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:22.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:22 smithi169 ceph-mon[35668]: osdmap e43: 6 total, 6 up, 6 in 2024-09-10T03:40:22.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.649+0000 7f8473d6e640 1 -- 172.21.15.112:0/2871363840 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f846c062df0 con 0x7f846c102c30 2024-09-10T03:40:22.650 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.649+0000 7f845effd640 1 -- 172.21.15.112:0/2871363840 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) ==== 75+0+1272 (secure 0 0 0) 0x7f846805ed20 con 0x7f846c102c30 2024-09-10T03:40:22.651 INFO:teuthology.orchestra.run.smithi112.stdout:e2 2024-09-10T03:40:22.651 INFO:teuthology.orchestra.run.smithi112.stdout:btime 2024-09-10T03:40:20:026721+0000 2024-09-10T03:40:22.651 INFO:teuthology.orchestra.run.smithi112.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-10T03:40:22.651 INFO:teuthology.orchestra.run.smithi112.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-10T03:40:22.651 INFO:teuthology.orchestra.run.smithi112.stdout:legacy client fscid: 1 2024-09-10T03:40:22.651 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:40:22.651 INFO:teuthology.orchestra.run.smithi112.stdout:Filesystem 'cephfs' (1) 2024-09-10T03:40:22.651 INFO:teuthology.orchestra.run.smithi112.stdout:fs_name cephfs 2024-09-10T03:40:22.651 INFO:teuthology.orchestra.run.smithi112.stdout:epoch 2 2024-09-10T03:40:22.651 INFO:teuthology.orchestra.run.smithi112.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-10T03:40:22.652 INFO:teuthology.orchestra.run.smithi112.stdout:created 2024-09-10T03:40:20.026648+0000 2024-09-10T03:40:22.652 INFO:teuthology.orchestra.run.smithi112.stdout:modified 2024-09-10T03:40:20.026648+0000 2024-09-10T03:40:22.652 INFO:teuthology.orchestra.run.smithi112.stdout:tableserver 0 2024-09-10T03:40:22.652 INFO:teuthology.orchestra.run.smithi112.stdout:root 0 2024-09-10T03:40:22.652 INFO:teuthology.orchestra.run.smithi112.stdout:session_timeout 60 2024-09-10T03:40:22.652 INFO:teuthology.orchestra.run.smithi112.stdout:session_autoclose 300 2024-09-10T03:40:22.652 INFO:teuthology.orchestra.run.smithi112.stdout:max_file_size 1099511627776 2024-09-10T03:40:22.652 INFO:teuthology.orchestra.run.smithi112.stdout:max_xattr_size 65536 2024-09-10T03:40:22.652 INFO:teuthology.orchestra.run.smithi112.stdout:required_client_features {} 2024-09-10T03:40:22.652 INFO:teuthology.orchestra.run.smithi112.stdout:last_failure 0 2024-09-10T03:40:22.652 INFO:teuthology.orchestra.run.smithi112.stdout:last_failure_osd_epoch 0 2024-09-10T03:40:22.653 INFO:teuthology.orchestra.run.smithi112.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-10T03:40:22.653 INFO:teuthology.orchestra.run.smithi112.stdout:max_mds 1 2024-09-10T03:40:22.653 INFO:teuthology.orchestra.run.smithi112.stdout:in 2024-09-10T03:40:22.653 INFO:teuthology.orchestra.run.smithi112.stdout:up {} 2024-09-10T03:40:22.653 INFO:teuthology.orchestra.run.smithi112.stdout:failed 2024-09-10T03:40:22.653 INFO:teuthology.orchestra.run.smithi112.stdout:damaged 2024-09-10T03:40:22.653 INFO:teuthology.orchestra.run.smithi112.stdout:stopped 2024-09-10T03:40:22.653 INFO:teuthology.orchestra.run.smithi112.stdout:data_pools [3] 2024-09-10T03:40:22.653 INFO:teuthology.orchestra.run.smithi112.stdout:metadata_pool 2 2024-09-10T03:40:22.653 INFO:teuthology.orchestra.run.smithi112.stdout:inline_data disabled 2024-09-10T03:40:22.653 INFO:teuthology.orchestra.run.smithi112.stdout:balancer 2024-09-10T03:40:22.653 INFO:teuthology.orchestra.run.smithi112.stdout:bal_rank_mask -1 2024-09-10T03:40:22.653 INFO:teuthology.orchestra.run.smithi112.stdout:standby_count_wanted 0 2024-09-10T03:40:22.653 INFO:teuthology.orchestra.run.smithi112.stdout:qdb_cluster leader: 0 members: 2024-09-10T03:40:22.654 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:40:22.654 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:40:22.654 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 2 2024-09-10T03:40:22.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.651+0000 7f8473d6e640 1 -- 172.21.15.112:0/2871363840 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f845806de30 msgr2=0x7f84580702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:22.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.651+0000 7f8473d6e640 1 --2- 172.21.15.112:0/2871363840 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f845806de30 0x7f84580702f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8460000c00 tx=0x7f8460005a20 comp rx=0 tx=0).stop 2024-09-10T03:40:22.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.651+0000 7f8473d6e640 1 -- 172.21.15.112:0/2871363840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f846c102c30 msgr2=0x7f846c197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:22.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.651+0000 7f8473d6e640 1 --2- 172.21.15.112:0/2871363840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f846c102c30 0x7f846c197a60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f846800b420 tx=0x7f846800b8e0 comp rx=0 tx=0).stop 2024-09-10T03:40:22.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.651+0000 7f8473d6e640 1 -- 172.21.15.112:0/2871363840 shutdown_connections 2024-09-10T03:40:22.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.651+0000 7f8473d6e640 1 --2- 172.21.15.112:0/2871363840 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f845806de30 0x7f84580702f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:22.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.651+0000 7f8473d6e640 1 --2- 172.21.15.112:0/2871363840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f846c102c30 0x7f846c197a60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:22.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.651+0000 7f8473d6e640 1 --2- 172.21.15.112:0/2871363840 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f846c0ffc50 0x7f846c197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:22.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.651+0000 7f8473d6e640 1 -- 172.21.15.112:0/2871363840 >> 172.21.15.112:0/2871363840 conn(0x7f846c0fb7a0 msgr2=0x7f846c1060f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:22.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.651+0000 7f8473d6e640 1 -- 172.21.15.112:0/2871363840 shutdown_connections 2024-09-10T03:40:22.655 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:22.651+0000 7f8473d6e640 1 -- 172.21.15.112:0/2871363840 wait complete. 2024-09-10T03:40:23.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:23 smithi112 ceph-mon[27737]: osdmap e44: 6 total, 6 up, 6 in 2024-09-10T03:40:23.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:23 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2871363840' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-10T03:40:23.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:23 smithi169 ceph-mon[35668]: osdmap e44: 6 total, 6 up, 6 in 2024-09-10T03:40:23.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:23 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/2871363840' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-10T03:40:23.737 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-10T03:40:23.748 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi112.front.sepia.ceph.com 2024-09-10T03:40:23.748 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-09-10T03:40:24.001 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:24.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:24 smithi169 ceph-mon[35668]: pgmap v176: 65 pgs: 1 creating+activating, 19 creating+peering, 29 active+clean, 16 unknown; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:24.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:24 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:24.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:24 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:24.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:24 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:24.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:24 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi169.wfmonc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-10T03:40:24.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:24 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi169.wfmonc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-10T03:40:24.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:24 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:40:24.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:24 smithi112 ceph-mon[27737]: pgmap v176: 65 pgs: 1 creating+activating, 19 creating+peering, 29 active+clean, 16 unknown; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:40:24.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:24 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:24.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:24 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:24.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:24 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:24.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:24 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi169.wfmonc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-10T03:40:24.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:24 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi169.wfmonc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-10T03:40:24.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:24 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:40:24.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.478+0000 7f9fac049640 1 -- 172.21.15.112:0/3033036547 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fa4101a80 msgr2=0x7f9fa4101f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:24.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.478+0000 7f9fac049640 1 --2- 172.21.15.112:0/3033036547 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fa4101a80 0x7f9fa4101f00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9f98009880 tx=0x7f9f9802a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:24.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.478+0000 7f9fac049640 1 -- 172.21.15.112:0/3033036547 shutdown_connections 2024-09-10T03:40:24.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.478+0000 7f9fac049640 1 --2- 172.21.15.112:0/3033036547 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fa4101a80 0x7f9fa4101f00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:24.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.478+0000 7f9fac049640 1 --2- 172.21.15.112:0/3033036547 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fa4100880 0x7f9fa4100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:24.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.478+0000 7f9fac049640 1 -- 172.21.15.112:0/3033036547 >> 172.21.15.112:0/3033036547 conn(0x7f9fa40fc010 msgr2=0x7f9fa40fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:24.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.478+0000 7f9fac049640 1 -- 172.21.15.112:0/3033036547 shutdown_connections 2024-09-10T03:40:24.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.478+0000 7f9fac049640 1 -- 172.21.15.112:0/3033036547 wait complete. 2024-09-10T03:40:24.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.479+0000 7f9fac049640 1 Processor -- start 2024-09-10T03:40:24.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.479+0000 7f9fac049640 1 -- start start 2024-09-10T03:40:24.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.479+0000 7f9fac049640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fa4100880 0x7f9fa41974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:24.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.479+0000 7f9fac049640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fa4101a80 0x7f9fa41979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:24.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.479+0000 7f9fac049640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa41096a0 con 0x7f9fa4101a80 2024-09-10T03:40:24.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.479+0000 7f9fac049640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa4109520 con 0x7f9fa4100880 2024-09-10T03:40:24.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.479+0000 7f9fa95e4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fa4101a80 0x7f9fa41979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:24.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.479+0000 7f9fa95e4640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fa4101a80 0x7f9fa41979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:60182/0 (socket says 172.21.15.112:60182) 2024-09-10T03:40:24.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.479+0000 7f9fa95e4640 1 -- 172.21.15.112:0/3563750372 learned_addr learned my addr 172.21.15.112:0/3563750372 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:24.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.479+0000 7f9fa9de5640 1 --2- 172.21.15.112:0/3563750372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fa4100880 0x7f9fa41974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:24.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.479+0000 7f9fa95e4640 1 -- 172.21.15.112:0/3563750372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fa4100880 msgr2=0x7f9fa41974b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:24.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.479+0000 7f9fa95e4640 1 --2- 172.21.15.112:0/3563750372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fa4100880 0x7f9fa41974b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:24.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.479+0000 7f9fa95e4640 1 -- 172.21.15.112:0/3563750372 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa4197fe0 con 0x7f9fa4101a80 2024-09-10T03:40:24.482 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.480+0000 7f9fa95e4640 1 --2- 172.21.15.112:0/3563750372 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fa4101a80 0x7f9fa41979f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9f98006010 tx=0x7f9f980048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:24.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.480+0000 7f9f96ffd640 1 -- 172.21.15.112:0/3563750372 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f9f9803f070 con 0x7f9fa4101a80 2024-09-10T03:40:24.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.480+0000 7f9fac049640 1 -- 172.21.15.112:0/3563750372 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fa419c9f0 con 0x7f9fa4101a80 2024-09-10T03:40:24.483 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.480+0000 7f9f96ffd640 1 -- 172.21.15.112:0/3563750372 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9f980044d0 con 0x7f9fa4101a80 2024-09-10T03:40:24.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.480+0000 7f9f96ffd640 1 -- 172.21.15.112:0/3563750372 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f9f98035030 con 0x7f9fa4101a80 2024-09-10T03:40:24.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.480+0000 7f9fac049640 1 -- 172.21.15.112:0/3563750372 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fa419cef0 con 0x7f9fa4101a80 2024-09-10T03:40:24.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.481+0000 7f9fac049640 1 -- 172.21.15.112:0/3563750372 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fa4100c80 con 0x7f9fa4101a80 2024-09-10T03:40:24.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.482+0000 7f9f96ffd640 1 -- 172.21.15.112:0/3563750372 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f9f98042070 con 0x7f9fa4101a80 2024-09-10T03:40:24.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.482+0000 7f9f96ffd640 1 --2- 172.21.15.112:0/3563750372 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f9f9006de30 0x7f9f900702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:24.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.482+0000 7f9f96ffd640 1 -- 172.21.15.112:0/3563750372 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5468+0+0 (secure 0 0 0) 0x7f9f980ae850 con 0x7f9fa4101a80 2024-09-10T03:40:24.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.482+0000 7f9fa9de5640 1 --2- 172.21.15.112:0/3563750372 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f9f9006de30 0x7f9f900702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:24.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.482+0000 7f9fa9de5640 1 --2- 172.21.15.112:0/3563750372 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f9f9006de30 0x7f9f900702f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9fa41018e0 tx=0x7f9fa0005f90 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:24.487 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.486+0000 7f9f96ffd640 1 -- 172.21.15.112:0/3563750372 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9f980800a0 con 0x7f9fa4101a80 2024-09-10T03:40:24.688 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:24.686+0000 7f9fac049640 1 -- 172.21.15.112:0/3563750372 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) -- 0x7f9fa419d3f0 con 0x7f9fa4101a80 2024-09-10T03:40:25.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:25.057+0000 7f9f96ffd640 1 -- 172.21.15.112:0/3563750372 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v5) ==== 105+0+0 (secure 0 0 0) 0x7f9f980848d0 con 0x7f9fa4101a80 2024-09-10T03:40:25.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:25.059+0000 7f9fac049640 1 -- 172.21.15.112:0/3563750372 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f9f9006de30 msgr2=0x7f9f900702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:25.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:25.059+0000 7f9fac049640 1 --2- 172.21.15.112:0/3563750372 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f9f9006de30 0x7f9f900702f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9fa41018e0 tx=0x7f9fa0005f90 comp rx=0 tx=0).stop 2024-09-10T03:40:25.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:25.059+0000 7f9fac049640 1 -- 172.21.15.112:0/3563750372 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fa4101a80 msgr2=0x7f9fa41979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:25.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:25.059+0000 7f9fac049640 1 --2- 172.21.15.112:0/3563750372 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fa4101a80 0x7f9fa41979f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9f98006010 tx=0x7f9f980048f0 comp rx=0 tx=0).stop 2024-09-10T03:40:25.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:25.059+0000 7f9fac049640 1 -- 172.21.15.112:0/3563750372 shutdown_connections 2024-09-10T03:40:25.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:25.059+0000 7f9fac049640 1 --2- 172.21.15.112:0/3563750372 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f9f9006de30 0x7f9f900702f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:25.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:25.059+0000 7f9fac049640 1 --2- 172.21.15.112:0/3563750372 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9fa4101a80 0x7f9fa41979f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:25.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:25.059+0000 7f9fac049640 1 --2- 172.21.15.112:0/3563750372 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9fa4100880 0x7f9fa41974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:25.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:25.059+0000 7f9fac049640 1 -- 172.21.15.112:0/3563750372 >> 172.21.15.112:0/3563750372 conn(0x7f9fa40fc010 msgr2=0x7f9fa406a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:25.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:25.059+0000 7f9fac049640 1 -- 172.21.15.112:0/3563750372 shutdown_connections 2024-09-10T03:40:25.061 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:25.059+0000 7f9fac049640 1 -- 172.21.15.112:0/3563750372 wait complete. 2024-09-10T03:40:25.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:25 smithi112 ceph-mon[27737]: Deploying daemon mds.cephfs.smithi169.wfmonc on smithi169 2024-09-10T03:40:25.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:25 smithi112 ceph-mon[27737]: mds.? [v2:172.21.15.112:6826/3404361455,v1:172.21.15.112:6827/3404361455] up:boot 2024-09-10T03:40:25.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:25 smithi112 ceph-mon[27737]: daemon mds.cephfs.smithi112.ifnhnl assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-09-10T03:40:25.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:25 smithi112 ceph-mon[27737]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-10T03:40:25.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:25 smithi112 ceph-mon[27737]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-10T03:40:25.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:25 smithi112 ceph-mon[27737]: Cluster is now healthy 2024-09-10T03:40:25.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:25 smithi112 ceph-mon[27737]: fsmap cephfs:0 1 up:standby 2024-09-10T03:40:25.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:25 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.ifnhnl"}]: dispatch 2024-09-10T03:40:25.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:25 smithi112 ceph-mon[27737]: fsmap cephfs:1 {0=cephfs.smithi112.ifnhnl=up:creating} 2024-09-10T03:40:25.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:25 smithi112 ceph-mon[27737]: daemon mds.cephfs.smithi112.ifnhnl is now active in filesystem cephfs as rank 0 2024-09-10T03:40:25.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:25 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3563750372' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-09-10T03:40:25.456 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:25 smithi169 ceph-mon[35668]: Deploying daemon mds.cephfs.smithi169.wfmonc on smithi169 2024-09-10T03:40:25.456 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:25 smithi169 ceph-mon[35668]: mds.? [v2:172.21.15.112:6826/3404361455,v1:172.21.15.112:6827/3404361455] up:boot 2024-09-10T03:40:25.456 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:25 smithi169 ceph-mon[35668]: daemon mds.cephfs.smithi112.ifnhnl assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-09-10T03:40:25.456 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:25 smithi169 ceph-mon[35668]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-10T03:40:25.457 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:25 smithi169 ceph-mon[35668]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-10T03:40:25.457 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:25 smithi169 ceph-mon[35668]: Cluster is now healthy 2024-09-10T03:40:25.457 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:25 smithi169 ceph-mon[35668]: fsmap cephfs:0 1 up:standby 2024-09-10T03:40:25.457 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:25 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.ifnhnl"}]: dispatch 2024-09-10T03:40:25.457 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:25 smithi169 ceph-mon[35668]: fsmap cephfs:1 {0=cephfs.smithi112.ifnhnl=up:creating} 2024-09-10T03:40:25.457 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:25 smithi169 ceph-mon[35668]: daemon mds.cephfs.smithi112.ifnhnl is now active in filesystem cephfs as rank 0 2024-09-10T03:40:25.457 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:25 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3563750372' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-09-10T03:40:25.735 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-10T03:40:25.745 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi112.front.sepia.ceph.com 2024-09-10T03:40:25.745 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-09-10T03:40:26.011 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:26.324 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:26 smithi112 ceph-mon[27737]: pgmap v177: 65 pgs: 1 creating+activating, 10 creating+peering, 50 active+clean, 4 unknown; 578 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 633 B/s wr, 2 op/s 2024-09-10T03:40:26.324 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:26 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3563750372' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-09-10T03:40:26.325 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:26 smithi112 ceph-mon[27737]: mds.? [v2:172.21.15.112:6826/3404361455,v1:172.21.15.112:6827/3404361455] up:active 2024-09-10T03:40:26.325 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:26 smithi112 ceph-mon[27737]: fsmap cephfs:1 {0=cephfs.smithi112.ifnhnl=up:active} 2024-09-10T03:40:26.325 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:26 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:26.325 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:26 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:26.325 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:26 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:26.325 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:26 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi112.uuenbo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-10T03:40:26.325 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:26 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi112.uuenbo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-10T03:40:26.325 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:26 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:40:26.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.504+0000 7fe6cfb12640 1 -- 172.21.15.112:0/2664949154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6c8102b60 msgr2=0x7fe6c8102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:26.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.504+0000 7fe6cfb12640 1 --2- 172.21.15.112:0/2664949154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6c8102b60 0x7fe6c8102fe0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009880 tx=0x7fe6b802a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:26.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.504+0000 7fe6cfb12640 1 -- 172.21.15.112:0/2664949154 shutdown_connections 2024-09-10T03:40:26.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.504+0000 7fe6cfb12640 1 --2- 172.21.15.112:0/2664949154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6c8102b60 0x7fe6c8102fe0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:26.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.504+0000 7fe6cfb12640 1 --2- 172.21.15.112:0/2664949154 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6c8101960 0x7fe6c8101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:26.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.504+0000 7fe6cfb12640 1 -- 172.21.15.112:0/2664949154 >> 172.21.15.112:0/2664949154 conn(0x7fe6c80fd0f0 msgr2=0x7fe6c80ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:26.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.504+0000 7fe6cfb12640 1 -- 172.21.15.112:0/2664949154 shutdown_connections 2024-09-10T03:40:26.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.504+0000 7fe6cfb12640 1 -- 172.21.15.112:0/2664949154 wait complete. 2024-09-10T03:40:26.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.504+0000 7fe6cfb12640 1 Processor -- start 2024-09-10T03:40:26.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.504+0000 7fe6cfb12640 1 -- start start 2024-09-10T03:40:26.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.504+0000 7fe6cfb12640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6c8101960 0x7fe6c8075f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:26.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.504+0000 7fe6cfb12640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6c8102b60 0x7fe6c80745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:26.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.504+0000 7fe6cfb12640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c810a780 con 0x7fe6c8101960 2024-09-10T03:40:26.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.504+0000 7fe6cfb12640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c810a600 con 0x7fe6c8102b60 2024-09-10T03:40:26.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.505+0000 7fe6cd8ae640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6c8101960 0x7fe6c8075f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:26.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.505+0000 7fe6cd8ae640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6c8101960 0x7fe6c8075f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:40016/0 (socket says 172.21.15.112:40016) 2024-09-10T03:40:26.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.505+0000 7fe6cd8ae640 1 -- 172.21.15.112:0/3994478911 learned_addr learned my addr 172.21.15.112:0/3994478911 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:26.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.505+0000 7fe6cd0ad640 1 --2- 172.21.15.112:0/3994478911 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6c8102b60 0x7fe6c80745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:26.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.505+0000 7fe6cd8ae640 1 -- 172.21.15.112:0/3994478911 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6c8102b60 msgr2=0x7fe6c80745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:26.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.505+0000 7fe6cd8ae640 1 --2- 172.21.15.112:0/3994478911 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6c8102b60 0x7fe6c80745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:26.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.505+0000 7fe6cd8ae640 1 -- 172.21.15.112:0/3994478911 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6c8074b20 con 0x7fe6c8101960 2024-09-10T03:40:26.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.505+0000 7fe6cd8ae640 1 --2- 172.21.15.112:0/3994478911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6c8101960 0x7fe6c8075f20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe6c400c8f0 tx=0x7fe6c400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:26.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.505+0000 7fe6beffd640 1 -- 172.21.15.112:0/3994478911 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe6c4004020 con 0x7fe6c8101960 2024-09-10T03:40:26.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.505+0000 7fe6beffd640 1 -- 172.21.15.112:0/3994478911 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe6c4007e10 con 0x7fe6c8101960 2024-09-10T03:40:26.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.505+0000 7fe6cfb12640 1 -- 172.21.15.112:0/3994478911 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6c8074e10 con 0x7fe6c8101960 2024-09-10T03:40:26.512 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.505+0000 7fe6beffd640 1 -- 172.21.15.112:0/3994478911 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe6c4003e10 con 0x7fe6c8101960 2024-09-10T03:40:26.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.505+0000 7fe6cfb12640 1 -- 172.21.15.112:0/3994478911 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6c80753a0 con 0x7fe6c8101960 2024-09-10T03:40:26.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.506+0000 7fe6cfb12640 1 -- 172.21.15.112:0/3994478911 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6c8101d60 con 0x7fe6c8101960 2024-09-10T03:40:26.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.507+0000 7fe6beffd640 1 -- 172.21.15.112:0/3994478911 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fe6c40041c0 con 0x7fe6c8101960 2024-09-10T03:40:26.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.507+0000 7fe6beffd640 1 --2- 172.21.15.112:0/3994478911 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fe6b406de30 0x7fe6b40702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:26.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.507+0000 7fe6beffd640 1 -- 172.21.15.112:0/3994478911 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5468+0+0 (secure 0 0 0) 0x7fe6c408dd10 con 0x7fe6c8101960 2024-09-10T03:40:26.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.507+0000 7fe6cd0ad640 1 --2- 172.21.15.112:0/3994478911 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fe6b406de30 0x7fe6b40702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:26.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.507+0000 7fe6cd0ad640 1 --2- 172.21.15.112:0/3994478911 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fe6b406de30 0x7fe6b40702f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8075ca0 tx=0x7fe6b8005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:26.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.512+0000 7fe6beffd640 1 -- 172.21.15.112:0/3994478911 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe6c4015330 con 0x7fe6c8101960 2024-09-10T03:40:26.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:26 smithi169 ceph-mon[35668]: pgmap v177: 65 pgs: 1 creating+activating, 10 creating+peering, 50 active+clean, 4 unknown; 578 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 633 B/s wr, 2 op/s 2024-09-10T03:40:26.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:26 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3563750372' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-09-10T03:40:26.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:26 smithi169 ceph-mon[35668]: mds.? [v2:172.21.15.112:6826/3404361455,v1:172.21.15.112:6827/3404361455] up:active 2024-09-10T03:40:26.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:26 smithi169 ceph-mon[35668]: fsmap cephfs:1 {0=cephfs.smithi112.ifnhnl=up:active} 2024-09-10T03:40:26.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:26 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:26.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:26 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:26.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:26 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:26.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:26 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi112.uuenbo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-10T03:40:26.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:26 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi112.uuenbo", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-10T03:40:26.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:26 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:40:26.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:26.718+0000 7fe6cfb12640 1 -- 172.21.15.112:0/3994478911 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fe6c8191590 con 0x7fe6c8101960 2024-09-10T03:40:27.070 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:27.070+0000 7fe6beffd640 1 -- 172.21.15.112:0/3994478911 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7fe6c405b8b0 con 0x7fe6c8101960 2024-09-10T03:40:27.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:27.072+0000 7fe6cfb12640 1 -- 172.21.15.112:0/3994478911 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fe6b406de30 msgr2=0x7fe6b40702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:27.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:27.072+0000 7fe6cfb12640 1 --2- 172.21.15.112:0/3994478911 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fe6b406de30 0x7fe6b40702f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8075ca0 tx=0x7fe6b8005a20 comp rx=0 tx=0).stop 2024-09-10T03:40:27.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:27.072+0000 7fe6cfb12640 1 -- 172.21.15.112:0/3994478911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6c8101960 msgr2=0x7fe6c8075f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:27.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:27.072+0000 7fe6cfb12640 1 --2- 172.21.15.112:0/3994478911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6c8101960 0x7fe6c8075f20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe6c400c8f0 tx=0x7fe6c400cdb0 comp rx=0 tx=0).stop 2024-09-10T03:40:27.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:27.072+0000 7fe6cfb12640 1 -- 172.21.15.112:0/3994478911 shutdown_connections 2024-09-10T03:40:27.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:27.072+0000 7fe6cfb12640 1 --2- 172.21.15.112:0/3994478911 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fe6b406de30 0x7fe6b40702f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:27.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:27.072+0000 7fe6cfb12640 1 --2- 172.21.15.112:0/3994478911 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe6c8102b60 0x7fe6c80745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:27.073 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:27.072+0000 7fe6cfb12640 1 --2- 172.21.15.112:0/3994478911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe6c8101960 0x7fe6c8075f20 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:27.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:27.072+0000 7fe6cfb12640 1 -- 172.21.15.112:0/3994478911 >> 172.21.15.112:0/3994478911 conn(0x7fe6c80fd0f0 msgr2=0x7fe6c8106010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:27.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:27.072+0000 7fe6cfb12640 1 -- 172.21.15.112:0/3994478911 shutdown_connections 2024-09-10T03:40:27.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:27.072+0000 7fe6cfb12640 1 -- 172.21.15.112:0/3994478911 wait complete. 2024-09-10T03:40:27.262 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:27 smithi112 ceph-mon[27737]: Deploying daemon mds.cephfs.smithi112.uuenbo on smithi112 2024-09-10T03:40:27.262 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:27 smithi112 ceph-mon[27737]: mds.? [v2:172.21.15.169:6824/4030590581,v1:172.21.15.169:6825/4030590581] up:boot 2024-09-10T03:40:27.262 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:27 smithi112 ceph-mon[27737]: fsmap cephfs:1 {0=cephfs.smithi112.ifnhnl=up:active} 1 up:standby 2024-09-10T03:40:27.262 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:27 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi169.wfmonc"}]: dispatch 2024-09-10T03:40:27.262 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:27 smithi112 ceph-mon[27737]: fsmap cephfs:1 {0=cephfs.smithi112.ifnhnl=up:active} 1 up:standby 2024-09-10T03:40:27.263 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:27 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3994478911' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-09-10T03:40:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:27 smithi169 ceph-mon[35668]: Deploying daemon mds.cephfs.smithi112.uuenbo on smithi112 2024-09-10T03:40:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:27 smithi169 ceph-mon[35668]: mds.? [v2:172.21.15.169:6824/4030590581,v1:172.21.15.169:6825/4030590581] up:boot 2024-09-10T03:40:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:27 smithi169 ceph-mon[35668]: fsmap cephfs:1 {0=cephfs.smithi112.ifnhnl=up:active} 1 up:standby 2024-09-10T03:40:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:27 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi169.wfmonc"}]: dispatch 2024-09-10T03:40:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:27 smithi169 ceph-mon[35668]: fsmap cephfs:1 {0=cephfs.smithi112.ifnhnl=up:active} 1 up:standby 2024-09-10T03:40:27.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:27 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3994478911' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-09-10T03:40:27.971 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-10T03:40:27.981 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi112.front.sepia.ceph.com 2024-09-10T03:40:27.981 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-09-10T03:40:28.170 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:28 smithi169 ceph-mon[35668]: pgmap v178: 65 pgs: 65 active+clean; 578 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 749 B/s wr, 4 op/s 2024-09-10T03:40:28.170 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:28 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3994478911' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-09-10T03:40:28.170 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:28 smithi169 ceph-mon[35668]: fsmap cephfs:1 {0=cephfs.smithi112.ifnhnl=up:active} 1 up:standby 2024-09-10T03:40:28.290 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:28 smithi112 ceph-mon[27737]: pgmap v178: 65 pgs: 65 active+clean; 578 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 749 B/s wr, 4 op/s 2024-09-10T03:40:28.291 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:28 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3994478911' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-09-10T03:40:28.291 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:28 smithi112 ceph-mon[27737]: fsmap cephfs:1 {0=cephfs.smithi112.ifnhnl=up:active} 1 up:standby 2024-09-10T03:40:28.472 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:29.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.108+0000 7f3af7e55640 1 -- 172.21.15.112:0/690414833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3af00fffe0 msgr2=0x7f3af01003e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:29.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.108+0000 7f3af7e55640 1 --2- 172.21.15.112:0/690414833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3af00fffe0 0x7f3af01003e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3ae000ad30 tx=0x7f3ae002b860 comp rx=0 tx=0).stop 2024-09-10T03:40:29.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.108+0000 7f3af7e55640 1 -- 172.21.15.112:0/690414833 shutdown_connections 2024-09-10T03:40:29.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.108+0000 7f3af7e55640 1 --2- 172.21.15.112:0/690414833 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3af01011e0 0x7f3af0101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:29.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.108+0000 7f3af7e55640 1 --2- 172.21.15.112:0/690414833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3af00fffe0 0x7f3af01003e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:29.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.108+0000 7f3af7e55640 1 -- 172.21.15.112:0/690414833 >> 172.21.15.112:0/690414833 conn(0x7f3af00fb770 msgr2=0x7f3af00fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:29.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.108+0000 7f3af7e55640 1 -- 172.21.15.112:0/690414833 shutdown_connections 2024-09-10T03:40:29.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.108+0000 7f3af7e55640 1 -- 172.21.15.112:0/690414833 wait complete. 2024-09-10T03:40:29.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.108+0000 7f3af7e55640 1 Processor -- start 2024-09-10T03:40:29.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.108+0000 7f3af7e55640 1 -- start start 2024-09-10T03:40:29.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.109+0000 7f3af7e55640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3af00fffe0 0x7f3af0195280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:29.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.109+0000 7f3af7e55640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3af01011e0 0x7f3af01957c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:29.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.109+0000 7f3af7e55640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3af0108590 con 0x7f3af01011e0 2024-09-10T03:40:29.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.109+0000 7f3af7e55640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f3af0108410 con 0x7f3af00fffe0 2024-09-10T03:40:29.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.109+0000 7f3af53f0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3af01011e0 0x7f3af01957c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:29.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.109+0000 7f3af53f0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3af01011e0 0x7f3af01957c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:40042/0 (socket says 172.21.15.112:40042) 2024-09-10T03:40:29.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.109+0000 7f3af53f0640 1 -- 172.21.15.112:0/387104750 learned_addr learned my addr 172.21.15.112:0/387104750 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:29.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.109+0000 7f3af5bf1640 1 --2- 172.21.15.112:0/387104750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3af00fffe0 0x7f3af0195280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:29.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.109+0000 7f3af53f0640 1 -- 172.21.15.112:0/387104750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3af00fffe0 msgr2=0x7f3af0195280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:29.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.109+0000 7f3af53f0640 1 --2- 172.21.15.112:0/387104750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3af00fffe0 0x7f3af0195280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:29.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.109+0000 7f3af53f0640 1 -- 172.21.15.112:0/387104750 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3af0195e40 con 0x7f3af01011e0 2024-09-10T03:40:29.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.109+0000 7f3af53f0640 1 --2- 172.21.15.112:0/387104750 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3af01011e0 0x7f3af01957c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400e970 tx=0x7f3ae400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:29.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.109+0000 7f3aeeffd640 1 -- 172.21.15.112:0/387104750 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3ae400cac0 con 0x7f3af01011e0 2024-09-10T03:40:29.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.109+0000 7f3af7e55640 1 -- 172.21.15.112:0/387104750 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3af019a880 con 0x7f3af01011e0 2024-09-10T03:40:29.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.109+0000 7f3aeeffd640 1 -- 172.21.15.112:0/387104750 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3ae40054e0 con 0x7f3af01011e0 2024-09-10T03:40:29.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.109+0000 7f3aeeffd640 1 -- 172.21.15.112:0/387104750 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3ae4010430 con 0x7f3af01011e0 2024-09-10T03:40:29.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.110+0000 7f3af7e55640 1 -- 172.21.15.112:0/387104750 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3af019ae90 con 0x7f3af01011e0 2024-09-10T03:40:29.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.111+0000 7f3af7e55640 1 -- 172.21.15.112:0/387104750 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3af01003e0 con 0x7f3af01011e0 2024-09-10T03:40:29.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.111+0000 7f3aeeffd640 1 -- 172.21.15.112:0/387104750 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f3ae4004ec0 con 0x7f3af01011e0 2024-09-10T03:40:29.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.111+0000 7f3aeeffd640 1 --2- 172.21.15.112:0/387104750 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3ad806de30 0x7f3ad80702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:29.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.111+0000 7f3aeeffd640 1 -- 172.21.15.112:0/387104750 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5468+0+0 (secure 0 0 0) 0x7f3ae408e2a0 con 0x7f3af01011e0 2024-09-10T03:40:29.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.111+0000 7f3af5bf1640 1 --2- 172.21.15.112:0/387104750 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3ad806de30 0x7f3ad80702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:29.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.112+0000 7f3af5bf1640 1 --2- 172.21.15.112:0/387104750 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3ad806de30 0x7f3ad80702f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0000f80 tx=0x7f3ae0004a80 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:29.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.116+0000 7f3aeeffd640 1 -- 172.21.15.112:0/387104750 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3ae405be40 con 0x7f3af01011e0 2024-09-10T03:40:29.348 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:29.347+0000 7f3af7e55640 1 -- 172.21.15.112:0/387104750 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3af0062df0 con 0x7f3af01011e0 2024-09-10T03:40:30.074 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:30.073+0000 7f3aeeffd640 1 -- 172.21.15.112:0/387104750 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v9) ==== 168+0+0 (secure 0 0 0) 0x7f3ae405faf0 con 0x7f3af01011e0 2024-09-10T03:40:30.074 INFO:teuthology.orchestra.run.smithi112.stderr:inline data enabled 2024-09-10T03:40:30.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:30.075+0000 7f3af7e55640 1 -- 172.21.15.112:0/387104750 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3ad806de30 msgr2=0x7f3ad80702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:30.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:30.075+0000 7f3af7e55640 1 --2- 172.21.15.112:0/387104750 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3ad806de30 0x7f3ad80702f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0000f80 tx=0x7f3ae0004a80 comp rx=0 tx=0).stop 2024-09-10T03:40:30.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:30.076+0000 7f3af7e55640 1 -- 172.21.15.112:0/387104750 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3af01011e0 msgr2=0x7f3af01957c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:30.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:30.076+0000 7f3af7e55640 1 --2- 172.21.15.112:0/387104750 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3af01011e0 0x7f3af01957c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400e970 tx=0x7f3ae400ed70 comp rx=0 tx=0).stop 2024-09-10T03:40:30.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:30.076+0000 7f3af7e55640 1 -- 172.21.15.112:0/387104750 shutdown_connections 2024-09-10T03:40:30.077 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:30.076+0000 7f3af7e55640 1 --2- 172.21.15.112:0/387104750 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3ad806de30 0x7f3ad80702f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:30.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:30.076+0000 7f3af7e55640 1 --2- 172.21.15.112:0/387104750 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3af01011e0 0x7f3af01957c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:30.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:30.076+0000 7f3af7e55640 1 --2- 172.21.15.112:0/387104750 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3af00fffe0 0x7f3af0195280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:30.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:30.076+0000 7f3af7e55640 1 -- 172.21.15.112:0/387104750 >> 172.21.15.112:0/387104750 conn(0x7f3af00fb770 msgr2=0x7f3af01046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:30.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:30.076+0000 7f3af7e55640 1 -- 172.21.15.112:0/387104750 shutdown_connections 2024-09-10T03:40:30.078 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:30.076+0000 7f3af7e55640 1 -- 172.21.15.112:0/387104750 wait complete. 2024-09-10T03:40:30.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:30 smithi112 ceph-mon[27737]: pgmap v179: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-09-10T03:40:30.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:30 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/387104750' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T03:40:30.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:30 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:30.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:30 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:30.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:30 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:30.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:30 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi169.gcmhul", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-10T03:40:30.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:30 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi169.gcmhul", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-10T03:40:30.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:30 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:40:30.346 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:30 smithi169 ceph-mon[35668]: pgmap v179: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-09-10T03:40:30.346 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:30 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/387104750' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T03:40:30.346 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:30 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:30.347 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:30 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:30.347 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:30 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:30.347 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:30 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi169.gcmhul", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-10T03:40:30.347 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:30 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi169.gcmhul", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-10T03:40:30.347 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:30 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:40:30.731 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-10T03:40:30.742 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi112.front.sepia.ceph.com 2024-09-10T03:40:30.742 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- bash -c 'ceph fs dump' 2024-09-10T03:40:30.994 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:31.347 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:31 smithi112 ceph-mon[27737]: Deploying daemon mds.cephfs.smithi169.gcmhul on smithi169 2024-09-10T03:40:31.347 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:31 smithi112 ceph-mon[27737]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-09-10T03:40:31.347 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:31 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/387104750' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T03:40:31.347 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:31 smithi112 ceph-mon[27737]: mds.? [v2:172.21.15.112:6828/1752942945,v1:172.21.15.112:6829/1752942945] up:boot 2024-09-10T03:40:31.347 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:31 smithi112 ceph-mon[27737]: fsmap cephfs:1 {0=cephfs.smithi112.ifnhnl=up:active} 2 up:standby 2024-09-10T03:40:31.347 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:31 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.uuenbo"}]: dispatch 2024-09-10T03:40:31.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.426+0000 7fcf741ea640 1 -- 172.21.15.112:0/2078381543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf6c101a40 msgr2=0x7fcf6c101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:31.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.426+0000 7fcf741ea640 1 --2- 172.21.15.112:0/2078381543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf6c101a40 0x7fcf6c101ec0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009880 tx=0x7fcf6002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:31.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.427+0000 7fcf741ea640 1 -- 172.21.15.112:0/2078381543 shutdown_connections 2024-09-10T03:40:31.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.427+0000 7fcf741ea640 1 --2- 172.21.15.112:0/2078381543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf6c101a40 0x7fcf6c101ec0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:31.438 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.427+0000 7fcf741ea640 1 --2- 172.21.15.112:0/2078381543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf6c100840 0x7fcf6c100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.427+0000 7fcf741ea640 1 -- 172.21.15.112:0/2078381543 >> 172.21.15.112:0/2078381543 conn(0x7fcf6c0fbff0 msgr2=0x7fcf6c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.427+0000 7fcf741ea640 1 -- 172.21.15.112:0/2078381543 shutdown_connections 2024-09-10T03:40:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.427+0000 7fcf741ea640 1 -- 172.21.15.112:0/2078381543 wait complete. 2024-09-10T03:40:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.427+0000 7fcf741ea640 1 Processor -- start 2024-09-10T03:40:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.427+0000 7fcf741ea640 1 -- start start 2024-09-10T03:40:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.428+0000 7fcf741ea640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf6c100840 0x7fcf6c06a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.428+0000 7fcf741ea640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf6c101a40 0x7fcf6c19c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.428+0000 7fcf741ea640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c109700 con 0x7fcf6c101a40 2024-09-10T03:40:31.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.428+0000 7fcf741ea640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6c109580 con 0x7fcf6c100840 2024-09-10T03:40:31.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.428+0000 7fcf71785640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf6c101a40 0x7fcf6c19c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:31.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.428+0000 7fcf71785640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf6c101a40 0x7fcf6c19c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:40070/0 (socket says 172.21.15.112:40070) 2024-09-10T03:40:31.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.428+0000 7fcf71785640 1 -- 172.21.15.112:0/377762431 learned_addr learned my addr 172.21.15.112:0/377762431 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:31.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.428+0000 7fcf71f86640 1 --2- 172.21.15.112:0/377762431 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf6c100840 0x7fcf6c06a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:31.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.428+0000 7fcf71785640 1 -- 172.21.15.112:0/377762431 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf6c100840 msgr2=0x7fcf6c06a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:31.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.428+0000 7fcf71785640 1 --2- 172.21.15.112:0/377762431 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf6c100840 0x7fcf6c06a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:31.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.428+0000 7fcf71785640 1 -- 172.21.15.112:0/377762431 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf6c19c830 con 0x7fcf6c101a40 2024-09-10T03:40:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.428+0000 7fcf71785640 1 --2- 172.21.15.112:0/377762431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf6c101a40 0x7fcf6c19c240 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcf60006010 tx=0x7fcf600048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.428+0000 7fcf6affd640 1 -- 172.21.15.112:0/377762431 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fcf6003f070 con 0x7fcf6c101a40 2024-09-10T03:40:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.428+0000 7fcf6affd640 1 -- 172.21.15.112:0/377762431 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcf600044d0 con 0x7fcf6c101a40 2024-09-10T03:40:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.428+0000 7fcf741ea640 1 -- 172.21.15.112:0/377762431 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf6c19cac0 con 0x7fcf6c101a40 2024-09-10T03:40:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.428+0000 7fcf6affd640 1 -- 172.21.15.112:0/377762431 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fcf60035030 con 0x7fcf6c101a40 2024-09-10T03:40:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.428+0000 7fcf741ea640 1 -- 172.21.15.112:0/377762431 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf6c19d070 con 0x7fcf6c101a40 2024-09-10T03:40:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.429+0000 7fcf741ea640 1 -- 172.21.15.112:0/377762431 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf6c100c20 con 0x7fcf6c101a40 2024-09-10T03:40:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.430+0000 7fcf6affd640 1 -- 172.21.15.112:0/377762431 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fcf60042070 con 0x7fcf6c101a40 2024-09-10T03:40:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.430+0000 7fcf6affd640 1 --2- 172.21.15.112:0/377762431 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fcf5406de30 0x7fcf540702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.431+0000 7fcf71f86640 1 --2- 172.21.15.112:0/377762431 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fcf5406de30 0x7fcf540702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.431+0000 7fcf6affd640 1 -- 172.21.15.112:0/377762431 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5468+0+0 (secure 0 0 0) 0x7fcf600ae910 con 0x7fcf6c101a40 2024-09-10T03:40:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.431+0000 7fcf71f86640 1 --2- 172.21.15.112:0/377762431 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fcf5406de30 0x7fcf540702f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c1018a0 tx=0x7fcf5c00d040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:31.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.435+0000 7fcf6affd640 1 -- 172.21.15.112:0/377762431 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcf600800d0 con 0x7fcf6c101a40 2024-09-10T03:40:31.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:31 smithi169 ceph-mon[35668]: Deploying daemon mds.cephfs.smithi169.gcmhul on smithi169 2024-09-10T03:40:31.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:31 smithi169 ceph-mon[35668]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-09-10T03:40:31.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:31 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/387104750' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T03:40:31.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:31 smithi169 ceph-mon[35668]: mds.? [v2:172.21.15.112:6828/1752942945,v1:172.21.15.112:6829/1752942945] up:boot 2024-09-10T03:40:31.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:31 smithi169 ceph-mon[35668]: fsmap cephfs:1 {0=cephfs.smithi112.ifnhnl=up:active} 2 up:standby 2024-09-10T03:40:31.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:31 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.uuenbo"}]: dispatch 2024-09-10T03:40:31.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.635+0000 7fcf741ea640 1 -- 172.21.15.112:0/377762431 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fcf6c02d7c0 con 0x7fcf6c101a40 2024-09-10T03:40:31.636 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.636+0000 7fcf6affd640 1 -- 172.21.15.112:0/377762431 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 9 v9) ==== 75+0+1803 (secure 0 0 0) 0x7fcf60084900 con 0x7fcf6c101a40 2024-09-10T03:40:31.637 INFO:teuthology.orchestra.run.smithi112.stdout:e9 2024-09-10T03:40:31.637 INFO:teuthology.orchestra.run.smithi112.stdout:btime 2024-09-10T03:40:30:072797+0000 2024-09-10T03:40:31.637 INFO:teuthology.orchestra.run.smithi112.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-10T03:40:31.637 INFO:teuthology.orchestra.run.smithi112.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-10T03:40:31.637 INFO:teuthology.orchestra.run.smithi112.stdout:legacy client fscid: 1 2024-09-10T03:40:31.637 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:40:31.637 INFO:teuthology.orchestra.run.smithi112.stdout:Filesystem 'cephfs' (1) 2024-09-10T03:40:31.637 INFO:teuthology.orchestra.run.smithi112.stdout:fs_name cephfs 2024-09-10T03:40:31.637 INFO:teuthology.orchestra.run.smithi112.stdout:epoch 9 2024-09-10T03:40:31.638 INFO:teuthology.orchestra.run.smithi112.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-10T03:40:31.638 INFO:teuthology.orchestra.run.smithi112.stdout:created 2024-09-10T03:40:20.026648+0000 2024-09-10T03:40:31.638 INFO:teuthology.orchestra.run.smithi112.stdout:modified 2024-09-10T03:40:29.348357+0000 2024-09-10T03:40:31.638 INFO:teuthology.orchestra.run.smithi112.stdout:tableserver 0 2024-09-10T03:40:31.638 INFO:teuthology.orchestra.run.smithi112.stdout:root 0 2024-09-10T03:40:31.638 INFO:teuthology.orchestra.run.smithi112.stdout:session_timeout 60 2024-09-10T03:40:31.638 INFO:teuthology.orchestra.run.smithi112.stdout:session_autoclose 300 2024-09-10T03:40:31.638 INFO:teuthology.orchestra.run.smithi112.stdout:max_file_size 1099511627776 2024-09-10T03:40:31.638 INFO:teuthology.orchestra.run.smithi112.stdout:max_xattr_size 65536 2024-09-10T03:40:31.638 INFO:teuthology.orchestra.run.smithi112.stdout:required_client_features {} 2024-09-10T03:40:31.638 INFO:teuthology.orchestra.run.smithi112.stdout:last_failure 0 2024-09-10T03:40:31.638 INFO:teuthology.orchestra.run.smithi112.stdout:last_failure_osd_epoch 0 2024-09-10T03:40:31.638 INFO:teuthology.orchestra.run.smithi112.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-10T03:40:31.638 INFO:teuthology.orchestra.run.smithi112.stdout:max_mds 1 2024-09-10T03:40:31.639 INFO:teuthology.orchestra.run.smithi112.stdout:in 0 2024-09-10T03:40:31.639 INFO:teuthology.orchestra.run.smithi112.stdout:up {0=14568} 2024-09-10T03:40:31.639 INFO:teuthology.orchestra.run.smithi112.stdout:failed 2024-09-10T03:40:31.639 INFO:teuthology.orchestra.run.smithi112.stdout:damaged 2024-09-10T03:40:31.639 INFO:teuthology.orchestra.run.smithi112.stdout:stopped 2024-09-10T03:40:31.639 INFO:teuthology.orchestra.run.smithi112.stdout:data_pools [3] 2024-09-10T03:40:31.639 INFO:teuthology.orchestra.run.smithi112.stdout:metadata_pool 2 2024-09-10T03:40:31.639 INFO:teuthology.orchestra.run.smithi112.stdout:inline_data enabled 2024-09-10T03:40:31.639 INFO:teuthology.orchestra.run.smithi112.stdout:balancer 2024-09-10T03:40:31.639 INFO:teuthology.orchestra.run.smithi112.stdout:bal_rank_mask -1 2024-09-10T03:40:31.639 INFO:teuthology.orchestra.run.smithi112.stdout:standby_count_wanted 1 2024-09-10T03:40:31.639 INFO:teuthology.orchestra.run.smithi112.stdout:qdb_cluster leader: 14568 members: 14568 2024-09-10T03:40:31.639 INFO:teuthology.orchestra.run.smithi112.stdout:[mds.cephfs.smithi112.ifnhnl{0:14568} state up:active seq 2 addr [v2:172.21.15.112:6826/3404361455,v1:172.21.15.112:6827/3404361455] compat {c=[1],r=[1],i=[1fff]}] 2024-09-10T03:40:31.639 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:40:31.639 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:40:31.640 INFO:teuthology.orchestra.run.smithi112.stdout:Standby daemons: 2024-09-10T03:40:31.640 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:40:31.640 INFO:teuthology.orchestra.run.smithi112.stdout:[mds.cephfs.smithi112.uuenbo{-1:14588} state up:standby seq 1 addr [v2:172.21.15.112:6828/1752942945,v1:172.21.15.112:6829/1752942945] compat {c=[1],r=[1],i=[1fff]}] 2024-09-10T03:40:31.640 INFO:teuthology.orchestra.run.smithi112.stdout:[mds.cephfs.smithi169.wfmonc{-1:24295} state up:standby seq 1 addr [v2:172.21.15.169:6824/4030590581,v1:172.21.15.169:6825/4030590581] compat {c=[1],r=[1],i=[1fff]}] 2024-09-10T03:40:31.641 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 9 2024-09-10T03:40:31.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.638+0000 7fcf741ea640 1 -- 172.21.15.112:0/377762431 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fcf5406de30 msgr2=0x7fcf540702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:31.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.638+0000 7fcf741ea640 1 --2- 172.21.15.112:0/377762431 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fcf5406de30 0x7fcf540702f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c1018a0 tx=0x7fcf5c00d040 comp rx=0 tx=0).stop 2024-09-10T03:40:31.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.638+0000 7fcf741ea640 1 -- 172.21.15.112:0/377762431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf6c101a40 msgr2=0x7fcf6c19c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:31.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.638+0000 7fcf741ea640 1 --2- 172.21.15.112:0/377762431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf6c101a40 0x7fcf6c19c240 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcf60006010 tx=0x7fcf600048f0 comp rx=0 tx=0).stop 2024-09-10T03:40:31.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.638+0000 7fcf741ea640 1 -- 172.21.15.112:0/377762431 shutdown_connections 2024-09-10T03:40:31.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.638+0000 7fcf741ea640 1 --2- 172.21.15.112:0/377762431 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fcf5406de30 0x7fcf540702f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:31.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.638+0000 7fcf741ea640 1 --2- 172.21.15.112:0/377762431 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf6c101a40 0x7fcf6c19c240 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:31.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.638+0000 7fcf741ea640 1 --2- 172.21.15.112:0/377762431 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcf6c100840 0x7fcf6c06a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:31.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.638+0000 7fcf741ea640 1 -- 172.21.15.112:0/377762431 >> 172.21.15.112:0/377762431 conn(0x7fcf6c0fbff0 msgr2=0x7fcf6c104f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:31.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.638+0000 7fcf741ea640 1 -- 172.21.15.112:0/377762431 shutdown_connections 2024-09-10T03:40:31.641 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:31.638+0000 7fcf741ea640 1 -- 172.21.15.112:0/377762431 wait complete. 2024-09-10T03:40:32.189 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:32 smithi169 ceph-mon[35668]: pgmap v180: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-09-10T03:40:32.190 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:32 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/377762431' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-10T03:40:32.300 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:32 smithi112 ceph-mon[27737]: pgmap v180: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-09-10T03:40:32.300 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:32 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/377762431' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-10T03:40:32.329 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-09-10T03:40:32.595 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:33.055 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.053+0000 7fb0b5c84640 1 -- 172.21.15.112:0/2343577004 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0b0102b70 msgr2=0x7fb0b0102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:33.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.053+0000 7fb0b5c84640 1 --2- 172.21.15.112:0/2343577004 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0b0102b70 0x7fb0b0102ff0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0009880 tx=0x7fb0a002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:33.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.053+0000 7fb0b5c84640 1 -- 172.21.15.112:0/2343577004 shutdown_connections 2024-09-10T03:40:33.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.053+0000 7fb0b5c84640 1 --2- 172.21.15.112:0/2343577004 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0b0102b70 0x7fb0b0102ff0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:33.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.053+0000 7fb0b5c84640 1 --2- 172.21.15.112:0/2343577004 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0b0101970 0x7fb0b0101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:33.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.053+0000 7fb0b5c84640 1 -- 172.21.15.112:0/2343577004 >> 172.21.15.112:0/2343577004 conn(0x7fb0b00fd100 msgr2=0x7fb0b00ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:33.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.053+0000 7fb0b5c84640 1 -- 172.21.15.112:0/2343577004 shutdown_connections 2024-09-10T03:40:33.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.054+0000 7fb0b5c84640 1 -- 172.21.15.112:0/2343577004 wait complete. 2024-09-10T03:40:33.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.054+0000 7fb0b5c84640 1 Processor -- start 2024-09-10T03:40:33.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.054+0000 7fb0b5c84640 1 -- start start 2024-09-10T03:40:33.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.054+0000 7fb0b5c84640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0b0101970 0x7fb0b0197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:33.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.054+0000 7fb0b5c84640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0b0102b70 0x7fb0b01979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:33.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.054+0000 7fb0b5c84640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b010a790 con 0x7fb0b0102b70 2024-09-10T03:40:33.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.054+0000 7fb0b5c84640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b010a610 con 0x7fb0b0101970 2024-09-10T03:40:33.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.055+0000 7fb0af7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0b0101970 0x7fb0b0197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:33.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.055+0000 7fb0aeffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0b0102b70 0x7fb0b01979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:33.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.055+0000 7fb0af7fe640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0b0101970 0x7fb0b0197460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.112:59498/0 (socket says 172.21.15.112:59498) 2024-09-10T03:40:33.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.055+0000 7fb0af7fe640 1 -- 172.21.15.112:0/1374899461 learned_addr learned my addr 172.21.15.112:0/1374899461 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:33.057 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.055+0000 7fb0aeffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0b0102b70 0x7fb0b01979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:40086/0 (socket says 172.21.15.112:40086) 2024-09-10T03:40:33.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.055+0000 7fb0af7fe640 1 -- 172.21.15.112:0/1374899461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0b0102b70 msgr2=0x7fb0b01979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:33.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.055+0000 7fb0af7fe640 1 --2- 172.21.15.112:0/1374899461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0b0102b70 0x7fb0b01979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:33.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.055+0000 7fb0af7fe640 1 -- 172.21.15.112:0/1374899461 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b0198020 con 0x7fb0b0101970 2024-09-10T03:40:33.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.055+0000 7fb0aeffd640 1 --2- 172.21.15.112:0/1374899461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0b0102b70 0x7fb0b01979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T03:40:33.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.055+0000 7fb0af7fe640 1 --2- 172.21.15.112:0/1374899461 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0b0101970 0x7fb0b0197460 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb09800d6f0 tx=0x7fb09800dbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:33.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.055+0000 7fb0acff9640 1 -- 172.21.15.112:0/1374899461 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb098017070 con 0x7fb0b0101970 2024-09-10T03:40:33.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.055+0000 7fb0acff9640 1 -- 172.21.15.112:0/1374899461 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb09800de00 con 0x7fb0b0101970 2024-09-10T03:40:33.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.055+0000 7fb0b5c84640 1 -- 172.21.15.112:0/1374899461 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b019ca60 con 0x7fb0b0101970 2024-09-10T03:40:33.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.055+0000 7fb0acff9640 1 -- 172.21.15.112:0/1374899461 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb098003bc0 con 0x7fb0b0101970 2024-09-10T03:40:33.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.056+0000 7fb0b5c84640 1 -- 172.21.15.112:0/1374899461 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b019cfc0 con 0x7fb0b0101970 2024-09-10T03:40:33.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.057+0000 7fb0b5c84640 1 -- 172.21.15.112:0/1374899461 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0b010af70 con 0x7fb0b0101970 2024-09-10T03:40:33.058 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.057+0000 7fb0acff9640 1 -- 172.21.15.112:0/1374899461 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fb098009a30 con 0x7fb0b0101970 2024-09-10T03:40:33.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.057+0000 7fb0acff9640 1 --2- 172.21.15.112:0/1374899461 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb09c06de30 0x7fb09c0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:33.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.058+0000 7fb0acff9640 1 -- 172.21.15.112:0/1374899461 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5468+0+0 (secure 0 0 0) 0x7fb09808d6d0 con 0x7fb0b0101970 2024-09-10T03:40:33.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.058+0000 7fb0aeffd640 1 --2- 172.21.15.112:0/1374899461 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb09c06de30 0x7fb09c0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:33.059 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.059+0000 7fb0aeffd640 1 --2- 172.21.15.112:0/1374899461 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb09c06de30 0x7fb09c0702f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0000c00 tx=0x7fb0a0005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:33.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.062+0000 7fb0acff9640 1 -- 172.21.15.112:0/1374899461 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb09805ef20 con 0x7fb0b0101970 2024-09-10T03:40:33.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.293+0000 7fb0b5c84640 1 -- 172.21.15.112:0/1374899461 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0b002db70 con 0x7fb0b0101970 2024-09-10T03:40:33.294 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.294+0000 7fb0acff9640 1 -- 172.21.15.112:0/1374899461 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+5327 (secure 0 0 0) 0x7fb0b002db70 con 0x7fb0b0101970 2024-09-10T03:40:33.294 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 10 2024-09-10T03:40:33.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.296+0000 7fb0b5c84640 1 -- 172.21.15.112:0/1374899461 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb09c06de30 msgr2=0x7fb09c0702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:33.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.296+0000 7fb0b5c84640 1 --2- 172.21.15.112:0/1374899461 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb09c06de30 0x7fb09c0702f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0000c00 tx=0x7fb0a0005a20 comp rx=0 tx=0).stop 2024-09-10T03:40:33.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.296+0000 7fb0b5c84640 1 -- 172.21.15.112:0/1374899461 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0b0101970 msgr2=0x7fb0b0197460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:33.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.296+0000 7fb0b5c84640 1 --2- 172.21.15.112:0/1374899461 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0b0101970 0x7fb0b0197460 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb09800d6f0 tx=0x7fb09800dbb0 comp rx=0 tx=0).stop 2024-09-10T03:40:33.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.296+0000 7fb0b5c84640 1 -- 172.21.15.112:0/1374899461 shutdown_connections 2024-09-10T03:40:33.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.296+0000 7fb0b5c84640 1 --2- 172.21.15.112:0/1374899461 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb09c06de30 0x7fb09c0702f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:33.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.296+0000 7fb0b5c84640 1 --2- 172.21.15.112:0/1374899461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb0b0102b70 0x7fb0b01979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:33.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.296+0000 7fb0b5c84640 1 --2- 172.21.15.112:0/1374899461 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb0b0101970 0x7fb0b0197460 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:33.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.296+0000 7fb0b5c84640 1 -- 172.21.15.112:0/1374899461 >> 172.21.15.112:0/1374899461 conn(0x7fb0b00fd100 msgr2=0x7fb0b0106030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:33.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.296+0000 7fb0b5c84640 1 -- 172.21.15.112:0/1374899461 shutdown_connections 2024-09-10T03:40:33.297 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:33.296+0000 7fb0b5c84640 1 -- 172.21.15.112:0/1374899461 wait complete. 2024-09-10T03:40:33.305 INFO:teuthology.orchestra.run.smithi112.stdout:true 2024-09-10T03:40:33.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:33 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:33.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:33 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:33.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:33 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:33.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:33 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:33.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:33 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:33.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:33 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:40:33.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:33 smithi112 ceph-mon[27737]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-09-10T03:40:33.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:33 smithi112 ceph-mon[27737]: mds.? [v2:172.21.15.169:6826/1849337020,v1:172.21.15.169:6827/1849337020] up:boot 2024-09-10T03:40:33.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:33 smithi112 ceph-mon[27737]: fsmap cephfs:1 {0=cephfs.smithi112.ifnhnl=up:active} 3 up:standby 2024-09-10T03:40:33.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:33 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi169.gcmhul"}]: dispatch 2024-09-10T03:40:33.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:33 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/1374899461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-10T03:40:33.744 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:33 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:33.745 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:33 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:33.745 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:33 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:33.745 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:33 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:33.745 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:33 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:33.745 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:33 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:40:33.745 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:33 smithi169 ceph-mon[35668]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-09-10T03:40:33.745 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:33 smithi169 ceph-mon[35668]: mds.? [v2:172.21.15.169:6826/1849337020,v1:172.21.15.169:6827/1849337020] up:boot 2024-09-10T03:40:33.745 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:33 smithi169 ceph-mon[35668]: fsmap cephfs:1 {0=cephfs.smithi112.ifnhnl=up:active} 3 up:standby 2024-09-10T03:40:33.745 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:33 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi169.gcmhul"}]: dispatch 2024-09-10T03:40:33.745 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:33 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/1374899461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-10T03:40:34.162 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-09-10T03:40:34.752 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:35.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.229+0000 7f2a400ec640 1 -- 172.21.15.112:0/700184566 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a38100840 msgr2=0x7f2a38100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:35.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.229+0000 7f2a400ec640 1 --2- 172.21.15.112:0/700184566 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a38100840 0x7f2a38100c40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c009880 tx=0x7f2a2c02a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:35.232 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.229+0000 7f2a400ec640 1 -- 172.21.15.112:0/700184566 shutdown_connections 2024-09-10T03:40:35.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.229+0000 7f2a400ec640 1 --2- 172.21.15.112:0/700184566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a38101a40 0x7f2a38101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:35.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.229+0000 7f2a400ec640 1 --2- 172.21.15.112:0/700184566 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a38100840 0x7f2a38100c40 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:35.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.229+0000 7f2a400ec640 1 -- 172.21.15.112:0/700184566 >> 172.21.15.112:0/700184566 conn(0x7f2a380fbff0 msgr2=0x7f2a380fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:35.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.229+0000 7f2a400ec640 1 -- 172.21.15.112:0/700184566 shutdown_connections 2024-09-10T03:40:35.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.229+0000 7f2a400ec640 1 -- 172.21.15.112:0/700184566 wait complete. 2024-09-10T03:40:35.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.229+0000 7f2a400ec640 1 Processor -- start 2024-09-10T03:40:35.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.229+0000 7f2a400ec640 1 -- start start 2024-09-10T03:40:35.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a400ec640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a38100840 0x7f2a3806a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:35.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a400ec640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a38101a40 0x7f2a3819c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:35.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a400ec640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2a38109680 con 0x7f2a38100840 2024-09-10T03:40:35.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a400ec640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2a38109500 con 0x7f2a38101a40 2024-09-10T03:40:35.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a3de88640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a38100840 0x7f2a3806a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:35.233 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a3de88640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a38100840 0x7f2a3806a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:40106/0 (socket says 172.21.15.112:40106) 2024-09-10T03:40:35.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a3de88640 1 -- 172.21.15.112:0/1740305503 learned_addr learned my addr 172.21.15.112:0/1740305503 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:35.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a3d687640 1 --2- 172.21.15.112:0/1740305503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a38101a40 0x7f2a3819c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:35.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a3de88640 1 -- 172.21.15.112:0/1740305503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a38101a40 msgr2=0x7f2a3819c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:35.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a3de88640 1 --2- 172.21.15.112:0/1740305503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a38101a40 0x7f2a3819c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:35.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a3de88640 1 -- 172.21.15.112:0/1740305503 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3819c7e0 con 0x7f2a38100840 2024-09-10T03:40:35.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a3d687640 1 --2- 172.21.15.112:0/1740305503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a38101a40 0x7f2a3819c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T03:40:35.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a3de88640 1 --2- 172.21.15.112:0/1740305503 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a38100840 0x7f2a3806a620 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c006010 tx=0x7f2a2c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:35.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a2affd640 1 -- 172.21.15.112:0/1740305503 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2a2c03f070 con 0x7f2a38100840 2024-09-10T03:40:35.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a2affd640 1 -- 172.21.15.112:0/1740305503 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2a2c0044d0 con 0x7f2a38100840 2024-09-10T03:40:35.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a400ec640 1 -- 172.21.15.112:0/1740305503 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a3819ca70 con 0x7f2a38100840 2024-09-10T03:40:35.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a2affd640 1 -- 172.21.15.112:0/1740305503 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2a2c035030 con 0x7f2a38100840 2024-09-10T03:40:35.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.230+0000 7f2a400ec640 1 -- 172.21.15.112:0/1740305503 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a3819cfa0 con 0x7f2a38100840 2024-09-10T03:40:35.234 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.231+0000 7f2a400ec640 1 -- 172.21.15.112:0/1740305503 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a38100c40 con 0x7f2a38100840 2024-09-10T03:40:35.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.232+0000 7f2a2affd640 1 -- 172.21.15.112:0/1740305503 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f2a2c042070 con 0x7f2a38100840 2024-09-10T03:40:35.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.232+0000 7f2a2affd640 1 --2- 172.21.15.112:0/1740305503 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2a2406de30 0x7f2a240702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:35.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.233+0000 7f2a3d687640 1 --2- 172.21.15.112:0/1740305503 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2a2406de30 0x7f2a240702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:35.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.233+0000 7f2a2affd640 1 -- 172.21.15.112:0/1740305503 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7f2a2c0ae930 con 0x7f2a38100840 2024-09-10T03:40:35.235 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.233+0000 7f2a3d687640 1 --2- 172.21.15.112:0/1740305503 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2a2406de30 0x7f2a240702f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2a34005fd0 tx=0x7f2a34005ea0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:35.238 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.237+0000 7f2a2affd640 1 -- 172.21.15.112:0/1740305503 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2a2c07c3e0 con 0x7f2a38100840 2024-09-10T03:40:35.452 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:35 smithi112 ceph-mon[27737]: mds.? [v2:172.21.15.112:6828/1752942945,v1:172.21.15.112:6829/1752942945] up:standby 2024-09-10T03:40:35.452 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:35 smithi112 ceph-mon[27737]: mds.? [v2:172.21.15.169:6824/4030590581,v1:172.21.15.169:6825/4030590581] up:standby 2024-09-10T03:40:35.452 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:35 smithi112 ceph-mon[27737]: Dropping low affinity active daemon mds.cephfs.smithi112.ifnhnl in favor of higher affinity standby. 2024-09-10T03:40:35.452 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:35 smithi112 ceph-mon[27737]: Replacing daemon mds.cephfs.smithi112.ifnhnl as rank 0 with standby daemon mds.cephfs.smithi112.uuenbo 2024-09-10T03:40:35.452 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:35 smithi112 ceph-mon[27737]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-10T03:40:35.452 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:35 smithi112 ceph-mon[27737]: fsmap cephfs:1 {0=cephfs.smithi112.ifnhnl=up:active} 3 up:standby 2024-09-10T03:40:35.452 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:35 smithi112 ceph-mon[27737]: osdmap e45: 6 total, 6 up, 6 in 2024-09-10T03:40:35.452 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:35 smithi112 ceph-mon[27737]: fsmap cephfs:1/1 {0=cephfs.smithi112.uuenbo=up:replay} 2 up:standby 2024-09-10T03:40:35.453 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:35.453 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:35.453 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:35.453 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:40:35.453 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:35 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:35.453 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:35 smithi112 ceph-mon[27737]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-10T03:40:35.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.480+0000 7f2a400ec640 1 -- 172.21.15.112:0/1740305503 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) -- 0x7f2a38062df0 con 0x7f2a38100840 2024-09-10T03:40:35.481 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.481+0000 7f2a2affd640 1 -- 172.21.15.112:0/1740305503 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v13) ==== 78+0+98 (secure 0 0 0) 0x7f2a2c080090 con 0x7f2a38100840 2024-09-10T03:40:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.483+0000 7f2a400ec640 1 -- 172.21.15.112:0/1740305503 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2a2406de30 msgr2=0x7f2a240702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.483+0000 7f2a400ec640 1 --2- 172.21.15.112:0/1740305503 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2a2406de30 0x7f2a240702f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2a34005fd0 tx=0x7f2a34005ea0 comp rx=0 tx=0).stop 2024-09-10T03:40:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.483+0000 7f2a400ec640 1 -- 172.21.15.112:0/1740305503 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a38100840 msgr2=0x7f2a3806a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.483+0000 7f2a400ec640 1 --2- 172.21.15.112:0/1740305503 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a38100840 0x7f2a3806a620 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c006010 tx=0x7f2a2c0048f0 comp rx=0 tx=0).stop 2024-09-10T03:40:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.483+0000 7f2a400ec640 1 -- 172.21.15.112:0/1740305503 shutdown_connections 2024-09-10T03:40:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.483+0000 7f2a400ec640 1 --2- 172.21.15.112:0/1740305503 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2a2406de30 0x7f2a240702f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.483+0000 7f2a400ec640 1 --2- 172.21.15.112:0/1740305503 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2a38101a40 0x7f2a3819c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.483+0000 7f2a400ec640 1 --2- 172.21.15.112:0/1740305503 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2a38100840 0x7f2a3806a620 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.483+0000 7f2a400ec640 1 -- 172.21.15.112:0/1740305503 >> 172.21.15.112:0/1740305503 conn(0x7f2a380fbff0 msgr2=0x7f2a38104ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.483+0000 7f2a400ec640 1 -- 172.21.15.112:0/1740305503 shutdown_connections 2024-09-10T03:40:35.484 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:35.483+0000 7f2a400ec640 1 -- 172.21.15.112:0/1740305503 wait complete. 2024-09-10T03:40:35.493 INFO:teuthology.orchestra.run.smithi112.stdout:true 2024-09-10T03:40:35.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:35 smithi169 ceph-mon[35668]: mds.? [v2:172.21.15.112:6828/1752942945,v1:172.21.15.112:6829/1752942945] up:standby 2024-09-10T03:40:35.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:35 smithi169 ceph-mon[35668]: mds.? [v2:172.21.15.169:6824/4030590581,v1:172.21.15.169:6825/4030590581] up:standby 2024-09-10T03:40:35.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:35 smithi169 ceph-mon[35668]: Dropping low affinity active daemon mds.cephfs.smithi112.ifnhnl in favor of higher affinity standby. 2024-09-10T03:40:35.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:35 smithi169 ceph-mon[35668]: Replacing daemon mds.cephfs.smithi112.ifnhnl as rank 0 with standby daemon mds.cephfs.smithi112.uuenbo 2024-09-10T03:40:35.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:35 smithi169 ceph-mon[35668]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-10T03:40:35.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:35 smithi169 ceph-mon[35668]: fsmap cephfs:1 {0=cephfs.smithi112.ifnhnl=up:active} 3 up:standby 2024-09-10T03:40:35.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:35 smithi169 ceph-mon[35668]: osdmap e45: 6 total, 6 up, 6 in 2024-09-10T03:40:35.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:35 smithi169 ceph-mon[35668]: fsmap cephfs:1/1 {0=cephfs.smithi112.uuenbo=up:replay} 2 up:standby 2024-09-10T03:40:35.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:35.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:35.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:35.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:40:35.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:35 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:35.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:35 smithi169 ceph-mon[35668]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-10T03:40:36.106 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-09-10T03:40:36.120 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph fs dump --format=json 2024-09-10T03:40:36.490 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:36.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:36 smithi112 ceph-mon[27737]: mds.? [v2:172.21.15.112:6828/1752942945,v1:172.21.15.112:6829/1752942945] up:reconnect 2024-09-10T03:40:36.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:36 smithi112 ceph-mon[27737]: mds.? [v2:172.21.15.112:6826/2447738559,v1:172.21.15.112:6827/2447738559] up:boot 2024-09-10T03:40:36.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:36 smithi112 ceph-mon[27737]: fsmap cephfs:1/1 {0=cephfs.smithi112.uuenbo=up:reconnect} 3 up:standby 2024-09-10T03:40:36.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:36 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.ifnhnl"}]: dispatch 2024-09-10T03:40:36.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:36 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/1740305503' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-09-10T03:40:36.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:36 smithi169 ceph-mon[35668]: mds.? [v2:172.21.15.112:6828/1752942945,v1:172.21.15.112:6829/1752942945] up:reconnect 2024-09-10T03:40:36.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:36 smithi169 ceph-mon[35668]: mds.? [v2:172.21.15.112:6826/2447738559,v1:172.21.15.112:6827/2447738559] up:boot 2024-09-10T03:40:36.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:36 smithi169 ceph-mon[35668]: fsmap cephfs:1/1 {0=cephfs.smithi112.uuenbo=up:reconnect} 3 up:standby 2024-09-10T03:40:36.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:36 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.ifnhnl"}]: dispatch 2024-09-10T03:40:36.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:36 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/1740305503' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-09-10T03:40:37.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.225+0000 7f38e2d9f640 1 -- 172.21.15.112:0/227635949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f38dc1067e0 msgr2=0x7f38dc106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:37.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.225+0000 7f38e2d9f640 1 --2- 172.21.15.112:0/227635949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f38dc1067e0 0x7f38dc106bc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f38d0009880 tx=0x7f38d002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:37.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.225+0000 7f38e2d9f640 1 -- 172.21.15.112:0/227635949 shutdown_connections 2024-09-10T03:40:37.245 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.225+0000 7f38e2d9f640 1 --2- 172.21.15.112:0/227635949 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38dc1007e0 0x7f38dc100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:37.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.225+0000 7f38e2d9f640 1 --2- 172.21.15.112:0/227635949 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f38dc1067e0 0x7f38dc106bc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:37.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.225+0000 7f38e2d9f640 1 -- 172.21.15.112:0/227635949 >> 172.21.15.112:0/227635949 conn(0x7f38dc0fc4c0 msgr2=0x7f38dc0fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:37.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.225+0000 7f38e2d9f640 1 -- 172.21.15.112:0/227635949 shutdown_connections 2024-09-10T03:40:37.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.225+0000 7f38e2d9f640 1 -- 172.21.15.112:0/227635949 wait complete. 2024-09-10T03:40:37.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.226+0000 7f38e2d9f640 1 Processor -- start 2024-09-10T03:40:37.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.226+0000 7f38e2d9f640 1 -- start start 2024-09-10T03:40:37.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.226+0000 7f38e2d9f640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38dc1007e0 0x7f38dc19d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:37.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.226+0000 7f38e2d9f640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f38dc1067e0 0x7f38dc19dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:37.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.226+0000 7f38e2d9f640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f38dc10dd10 con 0x7f38dc1067e0 2024-09-10T03:40:37.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.226+0000 7f38e2d9f640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f38dc10db90 con 0x7f38dc1007e0 2024-09-10T03:40:37.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.226+0000 7f38dbfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f38dc1067e0 0x7f38dc19dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:37.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.226+0000 7f38dbfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f38dc1067e0 0x7f38dc19dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:46354/0 (socket says 172.21.15.112:46354) 2024-09-10T03:40:37.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.226+0000 7f38dbfff640 1 -- 172.21.15.112:0/67138783 learned_addr learned my addr 172.21.15.112:0/67138783 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:37.246 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.226+0000 7f38dbfff640 1 -- 172.21.15.112:0/67138783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38dc1007e0 msgr2=0x7f38dc19d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:37.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.226+0000 7f38dbfff640 1 --2- 172.21.15.112:0/67138783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38dc1007e0 0x7f38dc19d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:37.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.226+0000 7f38dbfff640 1 -- 172.21.15.112:0/67138783 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38dc1977f0 con 0x7f38dc1067e0 2024-09-10T03:40:37.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.226+0000 7f38dbfff640 1 --2- 172.21.15.112:0/67138783 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f38dc1067e0 0x7f38dc19dbb0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f38c800b420 tx=0x7f38c800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:37.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.226+0000 7f38d9ffb640 1 -- 172.21.15.112:0/67138783 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f38c8011630 con 0x7f38dc1067e0 2024-09-10T03:40:37.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.227+0000 7f38e2d9f640 1 -- 172.21.15.112:0/67138783 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38dc197ae0 con 0x7f38dc1067e0 2024-09-10T03:40:37.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.227+0000 7f38d9ffb640 1 -- 172.21.15.112:0/67138783 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f38c8016070 con 0x7f38dc1067e0 2024-09-10T03:40:37.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.227+0000 7f38d9ffb640 1 -- 172.21.15.112:0/67138783 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f38c8010d40 con 0x7f38dc1067e0 2024-09-10T03:40:37.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.227+0000 7f38e2d9f640 1 -- 172.21.15.112:0/67138783 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38dc198070 con 0x7f38dc1067e0 2024-09-10T03:40:37.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.228+0000 7f38e2d9f640 1 -- 172.21.15.112:0/67138783 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38dc101f20 con 0x7f38dc1067e0 2024-09-10T03:40:37.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.228+0000 7f38d9ffb640 1 -- 172.21.15.112:0/67138783 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f38c801b460 con 0x7f38dc1067e0 2024-09-10T03:40:37.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.228+0000 7f38d9ffb640 1 --2- 172.21.15.112:0/67138783 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f38cc06de30 0x7f38cc0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:37.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.229+0000 7f38d9ffb640 1 -- 172.21.15.112:0/67138783 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7f38c808d440 con 0x7f38dc1067e0 2024-09-10T03:40:37.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.229+0000 7f38e0b3b640 1 --2- 172.21.15.112:0/67138783 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f38cc06de30 0x7f38cc0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:37.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.229+0000 7f38e0b3b640 1 --2- 172.21.15.112:0/67138783 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f38cc06de30 0x7f38cc0702f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f38d00096c0 tx=0x7f38d0000bc0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:37.247 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.233+0000 7f38d9ffb640 1 -- 172.21.15.112:0/67138783 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f38c805af80 con 0x7f38dc1067e0 2024-09-10T03:40:37.444 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.443+0000 7f38e2d9f640 1 -- 172.21.15.112:0/67138783 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38dc100c40 con 0x7f38dc1067e0 2024-09-10T03:40:37.445 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.444+0000 7f38d9ffb640 1 -- 172.21.15.112:0/67138783 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5328 (secure 0 0 0) 0x7f38c805ec30 con 0x7f38dc1067e0 2024-09-10T03:40:37.445 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-10T03:40:37.445 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":15,"btime":"2024-09-10T03:40:37:362980+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14602,"name":"cephfs.smithi112.ifnhnl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6827/2447738559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":2447738559},{"type":"v1","addr":"172.21.15.112:6827","nonce":2447738559}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24295,"name":"cephfs.smithi169.wfmonc","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.169:6825/4030590581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":4030590581},{"type":"v1","addr":"172.21.15.169:6825","nonce":4030590581}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24307,"name":"cephfs.smithi169.gcmhul","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.169:6827/1849337020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1849337020},{"type":"v1","addr":"172.21.15.169:6827","nonce":1849337020}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":14}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-10T03:40:20.026648+0000","modified":"2024-09-10T03:40:37.362974+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":14588},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14588":{"gid":14588,"name":"cephfs.smithi112.uuenbo","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.112:6829/1752942945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":1752942945},{"type":"v1","addr":"172.21.15.112:6829","nonce":1752942945}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14588,"qdb_cluster":[14588]},"id":1}]} 2024-09-10T03:40:37.445 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 15 2024-09-10T03:40:37.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.446+0000 7f38e2d9f640 1 -- 172.21.15.112:0/67138783 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f38cc06de30 msgr2=0x7f38cc0702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:37.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.446+0000 7f38e2d9f640 1 --2- 172.21.15.112:0/67138783 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f38cc06de30 0x7f38cc0702f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f38d00096c0 tx=0x7f38d0000bc0 comp rx=0 tx=0).stop 2024-09-10T03:40:37.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.446+0000 7f38e2d9f640 1 -- 172.21.15.112:0/67138783 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f38dc1067e0 msgr2=0x7f38dc19dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:37.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.446+0000 7f38e2d9f640 1 --2- 172.21.15.112:0/67138783 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f38dc1067e0 0x7f38dc19dbb0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f38c800b420 tx=0x7f38c800b8e0 comp rx=0 tx=0).stop 2024-09-10T03:40:37.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.446+0000 7f38e2d9f640 1 -- 172.21.15.112:0/67138783 shutdown_connections 2024-09-10T03:40:37.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.446+0000 7f38e2d9f640 1 --2- 172.21.15.112:0/67138783 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f38cc06de30 0x7f38cc0702f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:37.447 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.446+0000 7f38e2d9f640 1 --2- 172.21.15.112:0/67138783 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f38dc1067e0 0x7f38dc19dbb0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:37.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.446+0000 7f38e2d9f640 1 --2- 172.21.15.112:0/67138783 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f38dc1007e0 0x7f38dc19d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:37.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.446+0000 7f38e2d9f640 1 -- 172.21.15.112:0/67138783 >> 172.21.15.112:0/67138783 conn(0x7f38dc0fc4c0 msgr2=0x7f38dc1045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:37.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.446+0000 7f38e2d9f640 1 -- 172.21.15.112:0/67138783 shutdown_connections 2024-09-10T03:40:37.448 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:37.446+0000 7f38e2d9f640 1 -- 172.21.15.112:0/67138783 wait complete. 2024-09-10T03:40:37.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:37 smithi169 ceph-mon[35668]: mds.? [v2:172.21.15.112:6828/1752942945,v1:172.21.15.112:6829/1752942945] up:rejoin 2024-09-10T03:40:37.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:37 smithi169 ceph-mon[35668]: mds.? [v2:172.21.15.169:6826/1849337020,v1:172.21.15.169:6827/1849337020] up:standby 2024-09-10T03:40:37.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:37 smithi169 ceph-mon[35668]: fsmap cephfs:1/1 {0=cephfs.smithi112.uuenbo=up:rejoin} 3 up:standby 2024-09-10T03:40:37.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:37 smithi169 ceph-mon[35668]: daemon mds.cephfs.smithi112.uuenbo is now active in filesystem cephfs as rank 0 2024-09-10T03:40:37.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:37 smithi169 ceph-mon[35668]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 408 B/s rd, 920 B/s wr, 1 op/s 2024-09-10T03:40:37.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:37 smithi112 ceph-mon[27737]: mds.? [v2:172.21.15.112:6828/1752942945,v1:172.21.15.112:6829/1752942945] up:rejoin 2024-09-10T03:40:37.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:37 smithi112 ceph-mon[27737]: mds.? [v2:172.21.15.169:6826/1849337020,v1:172.21.15.169:6827/1849337020] up:standby 2024-09-10T03:40:37.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:37 smithi112 ceph-mon[27737]: fsmap cephfs:1/1 {0=cephfs.smithi112.uuenbo=up:rejoin} 3 up:standby 2024-09-10T03:40:37.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:37 smithi112 ceph-mon[27737]: daemon mds.cephfs.smithi112.uuenbo is now active in filesystem cephfs as rank 0 2024-09-10T03:40:37.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:37 smithi112 ceph-mon[27737]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 408 B/s rd, 920 B/s wr, 1 op/s 2024-09-10T03:40:38.229 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 15, 'max_mds': 1, 'flags': 18} 2024-09-10T03:40:38.229 INFO:teuthology.run_tasks:Running task kclient... 2024-09-10T03:40:38.258 INFO:tasks.kclient:Mounting kernel clients... 2024-09-10T03:40:38.258 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-09-10T03:40:38.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:38.259 DEBUG:teuthology.orchestra.run.smithi112:> ip netns list 2024-09-10T03:40:38.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:38.276 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link delete ceph-brx 2024-09-10T03:40:38.343 INFO:teuthology.orchestra.run.smithi112.stderr:Cannot find device "ceph-brx" 2024-09-10T03:40:38.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T03:40:38.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:38.345 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list 2024-09-10T03:40:38.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:38.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link delete ceph-brx 2024-09-10T03:40:38.439 INFO:teuthology.orchestra.run.smithi169.stderr:Cannot find device "ceph-brx" 2024-09-10T03:40:38.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T03:40:38.441 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-09-10T03:40:38.441 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-10T03:40:38.441 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-10T03:40:38.441 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph fs ls 2024-09-10T03:40:38.464 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:38 smithi112 ceph-mon[27737]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-10T03:40:38.464 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:38 smithi112 ceph-mon[27737]: mds.? [v2:172.21.15.112:6828/1752942945,v1:172.21.15.112:6829/1752942945] up:active 2024-09-10T03:40:38.464 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:38 smithi112 ceph-mon[27737]: fsmap cephfs:1 {0=cephfs.smithi112.uuenbo=up:active} 3 up:standby 2024-09-10T03:40:38.464 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:38 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/67138783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-10T03:40:38.642 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:38 smithi169 ceph-mon[35668]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-10T03:40:38.643 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:38 smithi169 ceph-mon[35668]: mds.? [v2:172.21.15.112:6828/1752942945,v1:172.21.15.112:6829/1752942945] up:active 2024-09-10T03:40:38.643 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:38 smithi169 ceph-mon[35668]: fsmap cephfs:1 {0=cephfs.smithi112.uuenbo=up:active} 3 up:standby 2024-09-10T03:40:38.643 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:38 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/67138783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-10T03:40:39.040 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:39.475 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:39 smithi112 ceph-mon[27737]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T03:40:39.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:39 smithi169 ceph-mon[35668]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T03:40:40.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.100+0000 7f167554a640 1 -- 172.21.15.112:0/3408069205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16701033d0 msgr2=0x7f167010b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:40.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.100+0000 7f167554a640 1 --2- 172.21.15.112:0/3408069205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16701033d0 0x7f167010b8c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1658009880 tx=0x7f165802a530 comp rx=0 tx=0).stop 2024-09-10T03:40:40.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.100+0000 7f167554a640 1 -- 172.21.15.112:0/3408069205 shutdown_connections 2024-09-10T03:40:40.112 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.100+0000 7f167554a640 1 --2- 172.21.15.112:0/3408069205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16701033d0 0x7f167010b8c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:40.113 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.100+0000 7f167554a640 1 --2- 172.21.15.112:0/3408069205 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1670102ab0 0x7f1670102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:40.113 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.100+0000 7f167554a640 1 -- 172.21.15.112:0/3408069205 >> 172.21.15.112:0/3408069205 conn(0x7f16700fc4f0 msgr2=0x7f16700fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:40.113 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.100+0000 7f167554a640 1 -- 172.21.15.112:0/3408069205 shutdown_connections 2024-09-10T03:40:40.113 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.100+0000 7f167554a640 1 -- 172.21.15.112:0/3408069205 wait complete. 2024-09-10T03:40:40.113 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.100+0000 7f167554a640 1 Processor -- start 2024-09-10T03:40:40.114 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.100+0000 7f167554a640 1 -- start start 2024-09-10T03:40:40.114 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.101+0000 7f167554a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1670102ab0 0x7f167019d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:40.114 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.101+0000 7f167554a640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16701033d0 0x7f167019db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:40.114 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.101+0000 7f167554a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f167010edc0 con 0x7f1670102ab0 2024-09-10T03:40:40.114 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.101+0000 7f167554a640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f167010ec40 con 0x7f16701033d0 2024-09-10T03:40:40.114 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.101+0000 7f166effd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1670102ab0 0x7f167019d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:40.114 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.101+0000 7f166effd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1670102ab0 0x7f167019d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:46368/0 (socket says 172.21.15.112:46368) 2024-09-10T03:40:40.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.101+0000 7f166effd640 1 -- 172.21.15.112:0/4284659190 learned_addr learned my addr 172.21.15.112:0/4284659190 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:40.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.101+0000 7f166effd640 1 -- 172.21.15.112:0/4284659190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16701033d0 msgr2=0x7f167019db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:40.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.101+0000 7f166effd640 1 --2- 172.21.15.112:0/4284659190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16701033d0 0x7f167019db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:40.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.101+0000 7f166effd640 1 -- 172.21.15.112:0/4284659190 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1670197790 con 0x7f1670102ab0 2024-09-10T03:40:40.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.101+0000 7f166effd640 1 --2- 172.21.15.112:0/4284659190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1670102ab0 0x7f167019d610 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f166000ea30 tx=0x7f166000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:40.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.101+0000 7f1667fff640 1 -- 172.21.15.112:0/4284659190 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f166000cbe0 con 0x7f1670102ab0 2024-09-10T03:40:40.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.101+0000 7f1667fff640 1 -- 172.21.15.112:0/4284659190 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f166000cd80 con 0x7f1670102ab0 2024-09-10T03:40:40.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.101+0000 7f167554a640 1 -- 172.21.15.112:0/4284659190 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1670197a80 con 0x7f1670102ab0 2024-09-10T03:40:40.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.101+0000 7f1667fff640 1 -- 172.21.15.112:0/4284659190 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f166001a920 con 0x7f1670102ab0 2024-09-10T03:40:40.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.101+0000 7f167554a640 1 -- 172.21.15.112:0/4284659190 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1670198010 con 0x7f1670102ab0 2024-09-10T03:40:40.115 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.102+0000 7f167554a640 1 -- 172.21.15.112:0/4284659190 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f167006ac30 con 0x7f1670102ab0 2024-09-10T03:40:40.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.103+0000 7f1667fff640 1 -- 172.21.15.112:0/4284659190 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f166001aac0 con 0x7f1670102ab0 2024-09-10T03:40:40.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.103+0000 7f1667fff640 1 --2- 172.21.15.112:0/4284659190 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f165c06de90 0x7f165c070350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:40.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.103+0000 7f1667fff640 1 -- 172.21.15.112:0/4284659190 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7f166008e220 con 0x7f1670102ab0 2024-09-10T03:40:40.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.103+0000 7f166e7fc640 1 --2- 172.21.15.112:0/4284659190 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f165c06de90 0x7f165c070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:40.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.104+0000 7f166e7fc640 1 --2- 172.21.15.112:0/4284659190 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f165c06de90 0x7f165c070350 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f16580096c0 tx=0x7f1658000bc0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:40.116 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.108+0000 7f1667fff640 1 -- 172.21.15.112:0/4284659190 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1660016030 con 0x7f1670102ab0 2024-09-10T03:40:40.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.332+0000 7f167554a640 1 -- 172.21.15.112:0/4284659190 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1670198320 con 0x7f1670102ab0 2024-09-10T03:40:40.334 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.333+0000 7f1667fff640 1 -- 172.21.15.112:0/4284659190 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+83 (secure 0 0 0) 0x7f166005bde0 con 0x7f1670102ab0 2024-09-10T03:40:40.334 INFO:teuthology.orchestra.run.smithi112.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-09-10T03:40:40.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.335+0000 7f167554a640 1 -- 172.21.15.112:0/4284659190 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f165c06de90 msgr2=0x7f165c070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:40.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.335+0000 7f167554a640 1 --2- 172.21.15.112:0/4284659190 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f165c06de90 0x7f165c070350 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f16580096c0 tx=0x7f1658000bc0 comp rx=0 tx=0).stop 2024-09-10T03:40:40.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.335+0000 7f167554a640 1 -- 172.21.15.112:0/4284659190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1670102ab0 msgr2=0x7f167019d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:40.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.335+0000 7f167554a640 1 --2- 172.21.15.112:0/4284659190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1670102ab0 0x7f167019d610 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f166000ea30 tx=0x7f166000eef0 comp rx=0 tx=0).stop 2024-09-10T03:40:40.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.336+0000 7f167554a640 1 -- 172.21.15.112:0/4284659190 shutdown_connections 2024-09-10T03:40:40.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.336+0000 7f167554a640 1 --2- 172.21.15.112:0/4284659190 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f165c06de90 0x7f165c070350 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:40.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.336+0000 7f167554a640 1 --2- 172.21.15.112:0/4284659190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f16701033d0 0x7f167019db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:40.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.336+0000 7f167554a640 1 --2- 172.21.15.112:0/4284659190 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1670102ab0 0x7f167019d610 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:40.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.336+0000 7f167554a640 1 -- 172.21.15.112:0/4284659190 >> 172.21.15.112:0/4284659190 conn(0x7f16700fc4f0 msgr2=0x7f167010aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:40.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.336+0000 7f167554a640 1 -- 172.21.15.112:0/4284659190 shutdown_connections 2024-09-10T03:40:40.337 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:40.336+0000 7f167554a640 1 -- 172.21.15.112:0/4284659190 wait complete. 2024-09-10T03:40:41.064 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-10T03:40:41.064 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T03:40:41.064 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi112.front.sepia.ceph.com 2024-09-10T03:40:41.064 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-10T03:40:41.065 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-10T03:40:41.065 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-10T03:40:41.065 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-10T03:40:41.065 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T03:40:41.065 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-10T03:40:41.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:41.065 DEBUG:teuthology.orchestra.run.smithi112:> ip addr 2024-09-10T03:40:41.081 INFO:teuthology.orchestra.run.smithi112.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T03:40:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T03:40:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T03:40:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: valid_lft forever preferred_lft forever 2024-09-10T03:40:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: inet6 ::1/128 scope host 2024-09-10T03:40:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: valid_lft forever preferred_lft forever 2024-09-10T03:40:41.081 INFO:teuthology.orchestra.run.smithi112.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T03:40:41.081 INFO:teuthology.orchestra.run.smithi112.stdout: link/ether 0c:c4:7a:d8:a5:2c brd ff:ff:ff:ff:ff:ff 2024-09-10T03:40:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: altname enp7s0f0 2024-09-10T03:40:41.082 INFO:teuthology.orchestra.run.smithi112.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T03:40:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: link/ether 0c:c4:7a:d8:a5:2d brd ff:ff:ff:ff:ff:ff 2024-09-10T03:40:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: altname enp7s0f1 2024-09-10T03:40:41.082 INFO:teuthology.orchestra.run.smithi112.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T03:40:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: link/ether 0c:c4:7a:8f:cb:68 brd ff:ff:ff:ff:ff:ff 2024-09-10T03:40:41.082 INFO:teuthology.orchestra.run.smithi112.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T03:40:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: link/ether 0c:c4:7a:8f:cb:69 brd ff:ff:ff:ff:ff:ff 2024-09-10T03:40:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: inet 172.21.15.112/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-10T03:40:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: valid_lft 42290sec preferred_lft 42290sec 2024-09-10T03:40:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: inet6 fe80::ec4:7aff:fe8f:cb69/64 scope link noprefixroute 2024-09-10T03:40:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: valid_lft forever preferred_lft forever 2024-09-10T03:40:41.083 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-10T03:40:41.083 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T03:40:41.083 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2024-09-10T03:40:41.083 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link add name ceph-brx type bridge 2024-09-10T03:40:41.083 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip addr flush dev ceph-brx 2024-09-10T03:40:41.083 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link set ceph-brx up 2024-09-10T03:40:41.083 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-10T03:40:41.083 DEBUG:teuthology.orchestra.run.smithi112:> ') 2024-09-10T03:40:41.173 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T03:40:41.345 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:41.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:41.351 DEBUG:teuthology.orchestra.run.smithi112:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T03:40:41.375 INFO:teuthology.orchestra.run.smithi112.stdout:1 2024-09-10T03:40:41.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:41.377 DEBUG:teuthology.orchestra.run.smithi112:> route 2024-09-10T03:40:41.468 INFO:teuthology.orchestra.run.smithi112.stdout:Kernel IP routing table 2024-09-10T03:40:41.468 INFO:teuthology.orchestra.run.smithi112.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T03:40:41.468 INFO:teuthology.orchestra.run.smithi112.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-10T03:40:41.468 INFO:teuthology.orchestra.run.smithi112.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-10T03:40:41.469 INFO:teuthology.orchestra.run.smithi112.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T03:40:41.470 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T03:40:41.470 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2024-09-10T03:40:41.470 DEBUG:teuthology.orchestra.run.smithi112:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-10T03:40:41.470 DEBUG:teuthology.orchestra.run.smithi112:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-10T03:40:41.470 DEBUG:teuthology.orchestra.run.smithi112:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-10T03:40:41.470 DEBUG:teuthology.orchestra.run.smithi112:> ') 2024-09-10T03:40:41.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T03:40:41.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:41 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:41.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:41 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:41.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:41 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/4284659190' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-10T03:40:41.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:41 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:40:41.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:41 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:40:41.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:41 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:41.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:41 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:40:41.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:41 smithi112 ceph-mon[27737]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 7.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-10T03:40:41.618 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:41 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:41.618 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:41 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:41.618 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:41 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/4284659190' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-10T03:40:41.618 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:41 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:40:41.618 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:41 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:40:41.618 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:41 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:41.618 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:41 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:40:41.618 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:41 smithi169 ceph-mon[35668]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 7.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-10T03:40:41.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:41.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:41.933 DEBUG:teuthology.orchestra.run.smithi112:> ip netns list 2024-09-10T03:40:41.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:41.949 DEBUG:teuthology.orchestra.run.smithi112:> ip netns list-id 2024-09-10T03:40:42.004 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T03:40:42.004 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2024-09-10T03:40:42.004 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-10T03:40:42.004 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-10T03:40:42.004 DEBUG:teuthology.orchestra.run.smithi112:> ') 2024-09-10T03:40:42.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T03:40:42.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:42.128 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-10T03:40:42.128 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T03:40:42.128 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2024-09-10T03:40:42.128 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-10T03:40:42.128 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-10T03:40:42.128 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-10T03:40:42.128 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-10T03:40:42.128 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-10T03:40:42.129 DEBUG:teuthology.orchestra.run.smithi112:> ') 2024-09-10T03:40:42.179 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T03:40:42.266 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:42.273 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T03:40:42.273 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2024-09-10T03:40:42.273 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link set brx.0 up 2024-09-10T03:40:42.273 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link set dev brx.0 master ceph-brx 2024-09-10T03:40:42.273 DEBUG:teuthology.orchestra.run.smithi112:> ') 2024-09-10T03:40:42.321 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T03:40:42.343 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:42.350 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T03:40:42.350 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-10T03:40:42.365 INFO:teuthology.orchestra.run.smithi112.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-10T03:40:42.366 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T03:40:42.366 DEBUG:teuthology.orchestra.run.smithi112:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-10T03:40:42.421 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T03:40:42.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:42.422 DEBUG:teuthology.orchestra.run.smithi112:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T03:40:42.542 INFO:teuthology.orchestra.run.smithi112.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T03:40:42.542 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T03:40:42.542 INFO:teuthology.orchestra.run.smithi112.stdout:invalid new device string format 2024-09-10T03:40:42.758 INFO:teuthology.orchestra.run.smithi112.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T03:40:42.758 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T03:40:42.758 INFO:teuthology.orchestra.run.smithi112.stdout:invalid new device string format 2024-09-10T03:40:42.759 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: resolved to: "172.21.15.112:3300,172.21.15.169:3300" 2024-09-10T03:40:42.759 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T03:40:42.763 INFO:teuthology.orchestra.run.smithi112.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-10T03:40:42.764 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-10T03:40:42.764 INFO:teuthology.orchestra.run.smithi112.stdout:invalid new device string format 2024-09-10T03:40:42.764 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: resolved to: "172.21.15.112:3300,172.21.15.169:3300" 2024-09-10T03:40:42.764 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T03:40:42.764 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: resolved to: "172.21.15.112:6789,172.21.15.169:6789" 2024-09-10T03:40:42.764 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.112:6789,172.21.15.169:6789:/ 2024-09-10T03:40:42.764 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=513b3bc6-6f25-11ef-bcea-c7b262605968" will pass to kernel 2024-09-10T03:40:42.765 INFO:teuthology.orchestra.run.smithi112.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-10T03:40:42.765 INFO:teuthology.orchestra.run.smithi112.stdout: You just mounted a file system that supports labels which does not 2024-09-10T03:40:42.765 INFO:teuthology.orchestra.run.smithi112.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T03:40:42.765 INFO:teuthology.orchestra.run.smithi112.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T03:40:42.765 INFO:teuthology.orchestra.run.smithi112.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T03:40:42.767 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T03:40:42.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:42.767 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-10T03:40:42.796 DEBUG:teuthology.orchestra.run.smithi112:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T03:40:42.885 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T03:40:42.890 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:42.896 INFO:teuthology.orchestra.run.smithi112.stdout:client24321 2024-09-10T03:40:42.896 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T03:40:42.896 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T03:40:42.915 DEBUG:teuthology.orchestra.run.smithi112:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T03:40:43.011 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T03:40:43.018 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:43.025 INFO:teuthology.orchestra.run.smithi112.stdout:client24321 2024-09-10T03:40:43.025 DEBUG:teuthology.orchestra.run.smithi112:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T03:40:43.080 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T03:40:43.084 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:43.091 INFO:teuthology.orchestra.run.smithi112.stdout:client24321 2024-09-10T03:40:43.091 DEBUG:teuthology.orchestra.run.smithi112:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968.client24321/status') 2024-09-10T03:40:43.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T03:40:43.153 INFO:teuthology.orchestra.run.smithi112.stdout:instance: client.24321 (3)172.21.15.112:0/1367855686 2024-09-10T03:40:43.153 INFO:teuthology.orchestra.run.smithi112.stdout:blocklisted: false 2024-09-10T03:40:43.153 INFO:teuthology.orchestra.run.smithi112.stderr:0+1 records in 2024-09-10T03:40:43.153 INFO:teuthology.orchestra.run.smithi112.stderr:0+1 records out 2024-09-10T03:40:43.153 INFO:teuthology.orchestra.run.smithi112.stderr:72 bytes copied, 2.465e-05 s, 2.9 MB/s 2024-09-10T03:40:43.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:43.160 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-09-10T03:40:43.160 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-10T03:40:43.160 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-10T03:40:43.161 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph fs ls 2024-09-10T03:40:43.671 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:44.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:43 smithi112 ceph-mon[27737]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-10T03:40:44.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:43 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:44.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:43 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:44.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:43 smithi169 ceph-mon[35668]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-10T03:40:44.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:43 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:44.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:43 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:44.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.459+0000 7f41106b7640 1 -- 172.21.15.112:0/3629299106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4108101910 msgr2=0x7f4108101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:44.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.459+0000 7f41106b7640 1 --2- 172.21.15.112:0/3629299106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4108101910 0x7f4108101d70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f40f8009880 tx=0x7f40f802a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:44.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.459+0000 7f41106b7640 1 -- 172.21.15.112:0/3629299106 shutdown_connections 2024-09-10T03:40:44.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.459+0000 7f41106b7640 1 --2- 172.21.15.112:0/3629299106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4108101910 0x7f4108101d70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:44.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.459+0000 7f41106b7640 1 --2- 172.21.15.112:0/3629299106 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4108107910 0x7f4108107cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:44.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.459+0000 7f41106b7640 1 -- 172.21.15.112:0/3629299106 >> 172.21.15.112:0/3629299106 conn(0x7f41080fd5f0 msgr2=0x7f41080ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:44.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.459+0000 7f41106b7640 1 -- 172.21.15.112:0/3629299106 shutdown_connections 2024-09-10T03:40:44.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.459+0000 7f41106b7640 1 -- 172.21.15.112:0/3629299106 wait complete. 2024-09-10T03:40:44.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.460+0000 7f41106b7640 1 Processor -- start 2024-09-10T03:40:44.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.460+0000 7f41106b7640 1 -- start start 2024-09-10T03:40:44.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.460+0000 7f41106b7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4108101910 0x7f41080744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:44.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.460+0000 7f41106b7640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4108107910 0x7f4108074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:44.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.460+0000 7f41106b7640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f410810ee40 con 0x7f4108107910 2024-09-10T03:40:44.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.460+0000 7f41106b7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f410810ecc0 con 0x7f4108101910 2024-09-10T03:40:44.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.460+0000 7f410dc52640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4108107910 0x7f4108074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:44.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.460+0000 7f410dc52640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4108107910 0x7f4108074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:46386/0 (socket says 172.21.15.112:46386) 2024-09-10T03:40:44.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.460+0000 7f410dc52640 1 -- 172.21.15.112:0/3860411357 learned_addr learned my addr 172.21.15.112:0/3860411357 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:44.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.460+0000 7f410e453640 1 --2- 172.21.15.112:0/3860411357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4108101910 0x7f41080744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:44.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.461+0000 7f410dc52640 1 -- 172.21.15.112:0/3860411357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4108101910 msgr2=0x7f41080744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:44.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.461+0000 7f410dc52640 1 --2- 172.21.15.112:0/3860411357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4108101910 0x7f41080744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:44.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.461+0000 7f410dc52640 1 -- 172.21.15.112:0/3860411357 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4108074f70 con 0x7f4108107910 2024-09-10T03:40:44.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.461+0000 7f410e453640 1 --2- 172.21.15.112:0/3860411357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4108101910 0x7f41080744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T03:40:44.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.461+0000 7f410dc52640 1 --2- 172.21.15.112:0/3860411357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4108107910 0x7f4108074a30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f40f8031460 tx=0x7f40f80048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:44.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.461+0000 7f40ff7fe640 1 -- 172.21.15.112:0/3860411357 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f40f803f070 con 0x7f4108107910 2024-09-10T03:40:44.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.461+0000 7f40ff7fe640 1 -- 172.21.15.112:0/3860411357 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f40f80044d0 con 0x7f4108107910 2024-09-10T03:40:44.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.461+0000 7f41106b7640 1 -- 172.21.15.112:0/3860411357 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4108075200 con 0x7f4108107910 2024-09-10T03:40:44.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.461+0000 7f40ff7fe640 1 -- 172.21.15.112:0/3860411357 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f40f8035030 con 0x7f4108107910 2024-09-10T03:40:44.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.461+0000 7f41106b7640 1 -- 172.21.15.112:0/3860411357 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41081a3e10 con 0x7f4108107910 2024-09-10T03:40:44.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.462+0000 7f41106b7640 1 -- 172.21.15.112:0/3860411357 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4108103050 con 0x7f4108107910 2024-09-10T03:40:44.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.463+0000 7f40ff7fe640 1 -- 172.21.15.112:0/3860411357 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f40f8042070 con 0x7f4108107910 2024-09-10T03:40:44.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.463+0000 7f40ff7fe640 1 --2- 172.21.15.112:0/3860411357 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f40f406de30 0x7f40f40702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:44.471 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.463+0000 7f40ff7fe640 1 -- 172.21.15.112:0/3860411357 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7f40f80ae760 con 0x7f4108107910 2024-09-10T03:40:44.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.463+0000 7f410e453640 1 --2- 172.21.15.112:0/3860411357 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f40f406de30 0x7f40f40702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:44.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.464+0000 7f410e453640 1 --2- 172.21.15.112:0/3860411357 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f40f406de30 0x7f40f40702f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4104009930 tx=0x7f4104008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:44.472 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.468+0000 7f40ff7fe640 1 -- 172.21.15.112:0/3860411357 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f40f8004cf0 con 0x7f4108107910 2024-09-10T03:40:44.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.697+0000 7f41106b7640 1 -- 172.21.15.112:0/3860411357 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4108109ea0 con 0x7f4108107910 2024-09-10T03:40:44.698 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.697+0000 7f40ff7fe640 1 -- 172.21.15.112:0/3860411357 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+83 (secure 0 0 0) 0x7f40f807c2a0 con 0x7f4108107910 2024-09-10T03:40:44.699 INFO:teuthology.orchestra.run.smithi112.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-09-10T03:40:44.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.700+0000 7f41106b7640 1 -- 172.21.15.112:0/3860411357 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f40f406de30 msgr2=0x7f40f40702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:44.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.700+0000 7f41106b7640 1 --2- 172.21.15.112:0/3860411357 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f40f406de30 0x7f40f40702f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4104009930 tx=0x7f4104008040 comp rx=0 tx=0).stop 2024-09-10T03:40:44.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.700+0000 7f41106b7640 1 -- 172.21.15.112:0/3860411357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4108107910 msgr2=0x7f4108074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:44.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.700+0000 7f41106b7640 1 --2- 172.21.15.112:0/3860411357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4108107910 0x7f4108074a30 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f40f8031460 tx=0x7f40f80048f0 comp rx=0 tx=0).stop 2024-09-10T03:40:44.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.700+0000 7f41106b7640 1 -- 172.21.15.112:0/3860411357 shutdown_connections 2024-09-10T03:40:44.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.700+0000 7f41106b7640 1 --2- 172.21.15.112:0/3860411357 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f40f406de30 0x7f40f40702f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:44.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.700+0000 7f41106b7640 1 --2- 172.21.15.112:0/3860411357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4108107910 0x7f4108074a30 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:44.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.700+0000 7f41106b7640 1 --2- 172.21.15.112:0/3860411357 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4108101910 0x7f41080744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:44.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.700+0000 7f41106b7640 1 -- 172.21.15.112:0/3860411357 >> 172.21.15.112:0/3860411357 conn(0x7f41080fd5f0 msgr2=0x7f41080fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:44.701 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.700+0000 7f41106b7640 1 -- 172.21.15.112:0/3860411357 shutdown_connections 2024-09-10T03:40:44.702 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:44.700+0000 7f41106b7640 1 -- 172.21.15.112:0/3860411357 wait complete. 2024-09-10T03:40:45.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:44 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3860411357' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-10T03:40:45.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:44 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3860411357' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-10T03:40:45.452 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-10T03:40:45.452 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-10T03:40:45.452 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi169.front.sepia.ceph.com 2024-09-10T03:40:45.452 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-10T03:40:45.452 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-10T03:40:45.452 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-10T03:40:45.452 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-10T03:40:45.453 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-10T03:40:45.453 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-10T03:40:45.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:45.453 DEBUG:teuthology.orchestra.run.smithi169:> ip addr 2024-09-10T03:40:45.474 INFO:teuthology.orchestra.run.smithi169.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-10T03:40:45.475 INFO:teuthology.orchestra.run.smithi169.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-10T03:40:45.475 INFO:teuthology.orchestra.run.smithi169.stdout: inet 127.0.0.1/8 scope host lo 2024-09-10T03:40:45.475 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-09-10T03:40:45.475 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 ::1/128 scope host 2024-09-10T03:40:45.475 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-09-10T03:40:45.475 INFO:teuthology.orchestra.run.smithi169.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T03:40:45.475 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:d9:3b:b4 brd ff:ff:ff:ff:ff:ff 2024-09-10T03:40:45.475 INFO:teuthology.orchestra.run.smithi169.stdout: altname enp7s0f0 2024-09-10T03:40:45.475 INFO:teuthology.orchestra.run.smithi169.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T03:40:45.475 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:d9:3b:b5 brd ff:ff:ff:ff:ff:ff 2024-09-10T03:40:45.475 INFO:teuthology.orchestra.run.smithi169.stdout: altname enp7s0f1 2024-09-10T03:40:45.476 INFO:teuthology.orchestra.run.smithi169.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-10T03:40:45.476 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:88:7f:4c brd ff:ff:ff:ff:ff:ff 2024-09-10T03:40:45.476 INFO:teuthology.orchestra.run.smithi169.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-10T03:40:45.476 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:88:7f:4d brd ff:ff:ff:ff:ff:ff 2024-09-10T03:40:45.476 INFO:teuthology.orchestra.run.smithi169.stdout: inet 172.21.15.169/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-10T03:40:45.476 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft 42334sec preferred_lft 42334sec 2024-09-10T03:40:45.476 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 fe80::ec4:7aff:fe88:7f4d/64 scope link noprefixroute 2024-09-10T03:40:45.476 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2024-09-10T03:40:45.476 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-10T03:40:45.477 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T03:40:45.477 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-09-10T03:40:45.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link add name ceph-brx type bridge 2024-09-10T03:40:45.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip addr flush dev ceph-brx 2024-09-10T03:40:45.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link set ceph-brx up 2024-09-10T03:40:45.477 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-10T03:40:45.477 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-09-10T03:40:45.570 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T03:40:45.692 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:45.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:45.698 DEBUG:teuthology.orchestra.run.smithi169:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-10T03:40:45.723 INFO:teuthology.orchestra.run.smithi169.stdout:1 2024-09-10T03:40:45.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:45.725 DEBUG:teuthology.orchestra.run.smithi169:> route 2024-09-10T03:40:45.797 INFO:teuthology.orchestra.run.smithi169.stdout:Kernel IP routing table 2024-09-10T03:40:45.798 INFO:teuthology.orchestra.run.smithi169.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-10T03:40:45.798 INFO:teuthology.orchestra.run.smithi169.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-10T03:40:45.798 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-10T03:40:45.798 INFO:teuthology.orchestra.run.smithi169.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-10T03:40:45.798 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T03:40:45.798 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-09-10T03:40:45.798 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-10T03:40:45.798 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-10T03:40:45.799 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-10T03:40:45.799 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-09-10T03:40:45.886 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T03:40:46.077 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:46.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:45 smithi112 ceph-mon[27737]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 7.8 KiB/s rd, 194 B/s wr, 7 op/s 2024-09-10T03:40:46.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:46.083 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list 2024-09-10T03:40:46.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:46.098 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list-id 2024-09-10T03:40:46.153 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T03:40:46.154 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-09-10T03:40:46.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-10T03:40:46.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-09-10T03:40:46.154 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-09-10T03:40:46.241 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T03:40:46.249 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:45 smithi169 ceph-mon[35668]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 7.8 KiB/s rd, 194 B/s wr, 7 op/s 2024-09-10T03:40:46.265 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:46.271 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-09-10T03:40:46.271 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T03:40:46.272 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-09-10T03:40:46.272 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-09-10T03:40:46.272 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-10T03:40:46.272 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-09-10T03:40:46.272 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-09-10T03:40:46.272 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-09-10T03:40:46.272 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-09-10T03:40:46.320 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T03:40:46.396 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:46.402 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-10T03:40:46.402 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2024-09-10T03:40:46.402 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link set brx.0 up 2024-09-10T03:40:46.402 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link set dev brx.0 master ceph-brx 2024-09-10T03:40:46.402 DEBUG:teuthology.orchestra.run.smithi169:> ') 2024-09-10T03:40:46.451 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T03:40:46.473 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:46.480 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T03:40:46.480 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-10T03:40:46.495 INFO:teuthology.orchestra.run.smithi169.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-10T03:40:46.496 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-10T03:40:46.496 DEBUG:teuthology.orchestra.run.smithi169:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-10T03:40:46.550 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-10T03:40:46.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:46.551 DEBUG:teuthology.orchestra.run.smithi169:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T03:40:46.670 INFO:teuthology.orchestra.run.smithi169.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T03:40:46.670 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T03:40:46.670 INFO:teuthology.orchestra.run.smithi169.stdout:invalid new device string format 2024-09-10T03:40:46.799 INFO:teuthology.orchestra.run.smithi169.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T03:40:46.799 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T03:40:46.800 INFO:teuthology.orchestra.run.smithi169.stdout:invalid new device string format 2024-09-10T03:40:46.800 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: resolved to: "172.21.15.112:3300,172.21.15.169:3300" 2024-09-10T03:40:46.800 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T03:40:46.804 INFO:teuthology.orchestra.run.smithi169.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-10T03:40:46.804 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-10T03:40:46.804 INFO:teuthology.orchestra.run.smithi169.stdout:invalid new device string format 2024-09-10T03:40:46.804 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: resolved to: "172.21.15.112:3300,172.21.15.169:3300" 2024-09-10T03:40:46.805 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-10T03:40:46.805 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: resolved to: "172.21.15.112:6789,172.21.15.169:6789" 2024-09-10T03:40:46.805 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.112:6789,172.21.15.169:6789:/ 2024-09-10T03:40:46.805 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=513b3bc6-6f25-11ef-bcea-c7b262605968" will pass to kernel 2024-09-10T03:40:46.805 INFO:teuthology.orchestra.run.smithi169.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-10T03:40:46.805 INFO:teuthology.orchestra.run.smithi169.stdout: You just mounted a file system that supports labels which does not 2024-09-10T03:40:46.806 INFO:teuthology.orchestra.run.smithi169.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-10T03:40:46.806 INFO:teuthology.orchestra.run.smithi169.stdout: applications will generate AVC messages and not be allowed access to 2024-09-10T03:40:46.806 INFO:teuthology.orchestra.run.smithi169.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-10T03:40:46.807 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-10T03:40:46.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T03:40:46.808 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-10T03:40:46.833 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T03:40:46.922 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:46 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T03:40:46.924 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:46.929 INFO:teuthology.orchestra.run.smithi169.stdout:client24325 2024-09-10T03:40:46.930 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-10T03:40:46.930 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T03:40:46.946 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T03:40:47.034 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T03:40:47.036 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:47.042 INFO:teuthology.orchestra.run.smithi169.stdout:client24325 2024-09-10T03:40:47.042 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-10T03:40:47.090 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-10T03:40:47.092 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:47.097 INFO:teuthology.orchestra.run.smithi169.stdout:client24325 2024-09-10T03:40:47.098 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968.client24325/status') 2024-09-10T03:40:47.146 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-10T03:40:47.155 INFO:teuthology.orchestra.run.smithi169.stdout:instance: client.24325 (3)192.168.0.1:0/710835813 2024-09-10T03:40:47.155 INFO:teuthology.orchestra.run.smithi169.stdout:blocklisted: false 2024-09-10T03:40:47.156 INFO:teuthology.orchestra.run.smithi169.stderr:0+1 records in 2024-09-10T03:40:47.156 INFO:teuthology.orchestra.run.smithi169.stderr:0+1 records out 2024-09-10T03:40:47.156 INFO:teuthology.orchestra.run.smithi169.stderr:69 bytes copied, 2.0971e-05 s, 3.3 MB/s 2024-09-10T03:40:47.156 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-10T03:40:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-10T03:40:47.163 INFO:teuthology.run_tasks:Running task print... 2024-09-10T03:40:47.173 INFO:teuthology.task.print:**** done client 2024-09-10T03:40:47.173 INFO:teuthology.run_tasks:Running task parallel... 2024-09-10T03:40:47.183 INFO:teuthology.task.parallel:starting parallel... 2024-09-10T03:40:47.183 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-10T03:40:47.184 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-09-10T03:40:47.184 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi112.front.sepia.ceph.com 2024-09-10T03:40:47.185 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -e sha1=3a5e3f16720f432aea965c40a8f6eb480095ec6c -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-10T03:40:47.185 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-10T03:40:47.185 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-09-10T03:40:47.190 INFO:tasks.workunit:Pulling workunits from ref 3a5e3f16720f432aea965c40a8f6eb480095ec6c 2024-09-10T03:40:47.191 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-10T03:40:47.191 INFO:tasks.workunit:timeout=3h 2024-09-10T03:40:47.191 INFO:tasks.workunit:cleanup=True 2024-09-10T03:40:47.191 DEBUG:teuthology.orchestra.run.smithi112:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-10T03:40:47.209 INFO:teuthology.orchestra.run.smithi112.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-09-10T03:40:47.209 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-09-10T03:40:47.209 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 87h/135d Inode: 1 Links: 2 2024-09-10T03:40:47.209 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T03:40:47.209 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-09-10T03:40:47.209 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-09-10T03:40:47.209 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-10 03:40:24.055369880 +0000 2024-09-10T03:40:47.209 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-10 03:40:42.794075262 +0000 2024-09-10T03:40:47.209 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-09-10 03:40:24.055369880 +0000 2024-09-10T03:40:47.209 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-09-10T03:40:47.210 DEBUG:teuthology.orchestra.run.smithi112:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-09-10T03:40:47.279 DEBUG:teuthology.orchestra.run.smithi169:> stat -- /home/ubuntu/cephtest/mnt.1 2024-09-10T03:40:47.296 INFO:teuthology.orchestra.run.smithi169.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-09-10T03:40:47.297 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-09-10T03:40:47.297 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 81h/129d Inode: 1 Links: 3 2024-09-10T03:40:47.297 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T03:40:47.297 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-09-10T03:40:47.297 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-09-10T03:40:47.297 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-09-10 03:40:47.271952804 +0000 2024-09-10T03:40:47.297 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-09-10 03:40:47.271952804 +0000 2024-09-10T03:40:47.297 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-09-10 03:40:24.055369880 +0000 2024-09-10T03:40:47.297 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-09-10T03:40:47.297 DEBUG:teuthology.orchestra.run.smithi169:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-09-10T03:40:47.365 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3a5e3f16720f432aea965c40a8f6eb480095ec6c 2024-09-10T03:40:47.366 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 3a5e3f16720f432aea965c40a8f6eb480095ec6c 2024-09-10T03:40:47.384 INFO:tasks.workunit.client.0.smithi112.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-10T03:40:47.424 INFO:tasks.workunit.client.1.smithi169.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-09-10T03:40:47.460 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:47.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:47 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:47.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:47 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:47.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:47 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:40:47.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:47 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:40:47.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:47 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:47.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:47 smithi169 ceph-mon[35668]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 6.8 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-10T03:40:47.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:47 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:47.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:47 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:47.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:47 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:40:47.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:47 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:40:47.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:47 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:47.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:47 smithi112 ceph-mon[27737]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 6.8 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-10T03:40:47.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.941+0000 7f58c8630640 1 -- 172.21.15.112:0/2643979202 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58c0101990 msgr2=0x7f58c0101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:47.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.941+0000 7f58c8630640 1 --2- 172.21.15.112:0/2643979202 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58c0101990 0x7f58c0101d90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f58bc009880 tx=0x7f58bc02a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:47.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.941+0000 7f58c8630640 1 -- 172.21.15.112:0/2643979202 shutdown_connections 2024-09-10T03:40:47.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.941+0000 7f58c8630640 1 --2- 172.21.15.112:0/2643979202 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58c0102b90 0x7f58c0103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:47.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.941+0000 7f58c8630640 1 --2- 172.21.15.112:0/2643979202 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58c0101990 0x7f58c0101d90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:47.945 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.941+0000 7f58c8630640 1 -- 172.21.15.112:0/2643979202 >> 172.21.15.112:0/2643979202 conn(0x7f58c00fd140 msgr2=0x7f58c00ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:47.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.941+0000 7f58c8630640 1 -- 172.21.15.112:0/2643979202 shutdown_connections 2024-09-10T03:40:47.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.941+0000 7f58c8630640 1 -- 172.21.15.112:0/2643979202 wait complete. 2024-09-10T03:40:47.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.942+0000 7f58c8630640 1 Processor -- start 2024-09-10T03:40:47.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.942+0000 7f58c8630640 1 -- start start 2024-09-10T03:40:47.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.942+0000 7f58c8630640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58c0101990 0x7f58c0075f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:47.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.942+0000 7f58c8630640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58c0102b90 0x7f58c00745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:47.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.942+0000 7f58c8630640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f58c010a7d0 con 0x7f58c0101990 2024-09-10T03:40:47.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.942+0000 7f58c8630640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f58c010a650 con 0x7f58c0102b90 2024-09-10T03:40:47.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.942+0000 7f58c63cc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58c0101990 0x7f58c0075f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:47.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.942+0000 7f58c63cc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58c0101990 0x7f58c0075f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:34014/0 (socket says 172.21.15.112:34014) 2024-09-10T03:40:47.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.942+0000 7f58c63cc640 1 -- 172.21.15.112:0/484692071 learned_addr learned my addr 172.21.15.112:0/484692071 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:47.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.942+0000 7f58c5bcb640 1 --2- 172.21.15.112:0/484692071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58c0102b90 0x7f58c00745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:47.946 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.943+0000 7f58c63cc640 1 -- 172.21.15.112:0/484692071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58c0102b90 msgr2=0x7f58c00745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:47.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.943+0000 7f58c63cc640 1 --2- 172.21.15.112:0/484692071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58c0102b90 0x7f58c00745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:47.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.943+0000 7f58c63cc640 1 -- 172.21.15.112:0/484692071 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c0074b20 con 0x7f58c0101990 2024-09-10T03:40:47.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.943+0000 7f58c5bcb640 1 --2- 172.21.15.112:0/484692071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58c0102b90 0x7f58c00745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T03:40:47.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.943+0000 7f58c63cc640 1 --2- 172.21.15.112:0/484692071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58c0101990 0x7f58c0075f20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f58bc006010 tx=0x7f58bc0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:47.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.943+0000 7f58b77fe640 1 -- 172.21.15.112:0/484692071 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f58bc03f070 con 0x7f58c0101990 2024-09-10T03:40:47.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.943+0000 7f58c8630640 1 -- 172.21.15.112:0/484692071 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58c0074db0 con 0x7f58c0101990 2024-09-10T03:40:47.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.943+0000 7f58b77fe640 1 -- 172.21.15.112:0/484692071 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f58bc004a50 con 0x7f58c0101990 2024-09-10T03:40:47.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.943+0000 7f58b77fe640 1 -- 172.21.15.112:0/484692071 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f58bc035030 con 0x7f58c0101990 2024-09-10T03:40:47.947 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.943+0000 7f58c8630640 1 -- 172.21.15.112:0/484692071 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58c00752e0 con 0x7f58c0101990 2024-09-10T03:40:47.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.944+0000 7f58c8630640 1 -- 172.21.15.112:0/484692071 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58c0101d90 con 0x7f58c0101990 2024-09-10T03:40:47.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.945+0000 7f58b77fe640 1 -- 172.21.15.112:0/484692071 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f58bc042070 con 0x7f58c0101990 2024-09-10T03:40:47.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.945+0000 7f58b77fe640 1 --2- 172.21.15.112:0/484692071 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f58ac06de30 0x7f58ac0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:47.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.945+0000 7f58b77fe640 1 -- 172.21.15.112:0/484692071 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7f58bc0ae7a0 con 0x7f58c0101990 2024-09-10T03:40:47.948 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.945+0000 7f58c5bcb640 1 --2- 172.21.15.112:0/484692071 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f58ac06de30 0x7f58ac0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:47.949 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.945+0000 7f58c5bcb640 1 --2- 172.21.15.112:0/484692071 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f58ac06de30 0x7f58ac0702f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f58b00058c0 tx=0x7f58b0005850 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:47.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:47.949+0000 7f58b77fe640 1 -- 172.21.15.112:0/484692071 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f58bc004c00 con 0x7f58c0101990 2024-09-10T03:40:48.119 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:48.117+0000 7f58c8630640 1 -- 172.21.15.112:0/484692071 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) -- 0x7f58c01915e0 con 0x7f58c0101990 2024-09-10T03:40:48.119 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:48.117+0000 7f58b77fe640 1 -- 172.21.15.112:0/484692071 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v43)=0 v43) ==== 155+0+0 (secure 0 0 0) 0x7f58bc07c2e0 con 0x7f58c0101990 2024-09-10T03:40:48.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:48.119+0000 7f58c8630640 1 -- 172.21.15.112:0/484692071 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f58ac06de30 msgr2=0x7f58ac0702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:48.121 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:48.119+0000 7f58c8630640 1 --2- 172.21.15.112:0/484692071 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f58ac06de30 0x7f58ac0702f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f58b00058c0 tx=0x7f58b0005850 comp rx=0 tx=0).stop 2024-09-10T03:40:48.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:48.119+0000 7f58c8630640 1 -- 172.21.15.112:0/484692071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58c0101990 msgr2=0x7f58c0075f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:48.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:48.119+0000 7f58c8630640 1 --2- 172.21.15.112:0/484692071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58c0101990 0x7f58c0075f20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f58bc006010 tx=0x7f58bc0045e0 comp rx=0 tx=0).stop 2024-09-10T03:40:48.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:48.119+0000 7f58c8630640 1 -- 172.21.15.112:0/484692071 shutdown_connections 2024-09-10T03:40:48.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:48.119+0000 7f58c8630640 1 --2- 172.21.15.112:0/484692071 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f58ac06de30 0x7f58ac0702f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:48.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:48.119+0000 7f58c8630640 1 --2- 172.21.15.112:0/484692071 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f58c0102b90 0x7f58c00745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:48.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:48.119+0000 7f58c8630640 1 --2- 172.21.15.112:0/484692071 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58c0101990 0x7f58c0075f20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:48.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:48.119+0000 7f58c8630640 1 -- 172.21.15.112:0/484692071 >> 172.21.15.112:0/484692071 conn(0x7f58c00fd140 msgr2=0x7f58c01060a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:48.122 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:48.119+0000 7f58c8630640 1 -- 172.21.15.112:0/484692071 shutdown_connections 2024-09-10T03:40:48.123 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:48.119+0000 7f58c8630640 1 -- 172.21.15.112:0/484692071 wait complete. 2024-09-10T03:40:48.722 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -e sha1=3a5e3f16720f432aea965c40a8f6eb480095ec6c -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-10T03:40:49.017 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:49.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.483+0000 7f31c7f94640 1 -- 172.21.15.112:0/3994395427 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31c0100880 msgr2=0x7f31c0100c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:49.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.483+0000 7f31c7f94640 1 --2- 172.21.15.112:0/3994395427 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31c0100880 0x7f31c0100c80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f31bc009880 tx=0x7f31bc02a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:49.490 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.483+0000 7f31c7f94640 1 -- 172.21.15.112:0/3994395427 shutdown_connections 2024-09-10T03:40:49.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.483+0000 7f31c7f94640 1 --2- 172.21.15.112:0/3994395427 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31c0101a80 0x7f31c0101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:49.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.483+0000 7f31c7f94640 1 --2- 172.21.15.112:0/3994395427 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31c0100880 0x7f31c0100c80 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:49.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.483+0000 7f31c7f94640 1 -- 172.21.15.112:0/3994395427 >> 172.21.15.112:0/3994395427 conn(0x7f31c00fc010 msgr2=0x7f31c00fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:49.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.483+0000 7f31c7f94640 1 -- 172.21.15.112:0/3994395427 shutdown_connections 2024-09-10T03:40:49.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.483+0000 7f31c7f94640 1 -- 172.21.15.112:0/3994395427 wait complete. 2024-09-10T03:40:49.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.484+0000 7f31c7f94640 1 Processor -- start 2024-09-10T03:40:49.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.484+0000 7f31c7f94640 1 -- start start 2024-09-10T03:40:49.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.484+0000 7f31c7f94640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31c0100880 0x7f31c006adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:49.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.484+0000 7f31c7f94640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31c0101a80 0x7f31c019c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:49.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.484+0000 7f31c7f94640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f31c0109720 con 0x7f31c0100880 2024-09-10T03:40:49.491 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.484+0000 7f31c7f94640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f31c01095a0 con 0x7f31c0101a80 2024-09-10T03:40:49.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.484+0000 7f31c5d30640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31c0100880 0x7f31c006adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:49.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.484+0000 7f31c5d30640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31c0100880 0x7f31c006adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:34032/0 (socket says 172.21.15.112:34032) 2024-09-10T03:40:49.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.484+0000 7f31c5d30640 1 -- 172.21.15.112:0/1963127424 learned_addr learned my addr 172.21.15.112:0/1963127424 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:49.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.484+0000 7f31c5d30640 1 -- 172.21.15.112:0/1963127424 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31c0101a80 msgr2=0x7f31c019c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:49.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.484+0000 7f31c5d30640 1 --2- 172.21.15.112:0/1963127424 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31c0101a80 0x7f31c019c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:49.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.484+0000 7f31c5d30640 1 -- 172.21.15.112:0/1963127424 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c019c850 con 0x7f31c0100880 2024-09-10T03:40:49.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.484+0000 7f31c5d30640 1 --2- 172.21.15.112:0/1963127424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31c0100880 0x7f31c006adf0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f31bc006010 tx=0x7f31bc0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:49.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.485+0000 7f31b6ffd640 1 -- 172.21.15.112:0/1963127424 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f31bc03f070 con 0x7f31c0100880 2024-09-10T03:40:49.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.485+0000 7f31b6ffd640 1 -- 172.21.15.112:0/1963127424 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f31bc0044d0 con 0x7f31c0100880 2024-09-10T03:40:49.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.485+0000 7f31c7f94640 1 -- 172.21.15.112:0/1963127424 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31c019cae0 con 0x7f31c0100880 2024-09-10T03:40:49.492 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.485+0000 7f31b6ffd640 1 -- 172.21.15.112:0/1963127424 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f31bc035030 con 0x7f31c0100880 2024-09-10T03:40:49.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.485+0000 7f31c7f94640 1 -- 172.21.15.112:0/1963127424 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c019d090 con 0x7f31c0100880 2024-09-10T03:40:49.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.486+0000 7f31c7f94640 1 -- 172.21.15.112:0/1963127424 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31c0100c80 con 0x7f31c0100880 2024-09-10T03:40:49.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.486+0000 7f31b6ffd640 1 -- 172.21.15.112:0/1963127424 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f31bc042070 con 0x7f31c0100880 2024-09-10T03:40:49.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.487+0000 7f31b6ffd640 1 --2- 172.21.15.112:0/1963127424 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f31ac06de30 0x7f31ac0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:49.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.487+0000 7f31c552f640 1 --2- 172.21.15.112:0/1963127424 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f31ac06de30 0x7f31ac0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:49.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.487+0000 7f31b6ffd640 1 -- 172.21.15.112:0/1963127424 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7f31bc0ae900 con 0x7f31c0100880 2024-09-10T03:40:49.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.487+0000 7f31c552f640 1 --2- 172.21.15.112:0/1963127424 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f31ac06de30 0x7f31ac0702f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f31c006ab70 tx=0x7f31b0006c90 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:49.493 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.491+0000 7f31b6ffd640 1 -- 172.21.15.112:0/1963127424 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f31bc07c3b0 con 0x7f31c0100880 2024-09-10T03:40:49.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.657+0000 7f31c7f94640 1 -- 172.21.15.112:0/1963127424 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) -- 0x7f31c0190530 con 0x7f31c0100880 2024-09-10T03:40:49.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.658+0000 7f31b6ffd640 1 -- 172.21.15.112:0/1963127424 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v43)=0 v43) ==== 163+0+0 (secure 0 0 0) 0x7f31c0190530 con 0x7f31c0100880 2024-09-10T03:40:49.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.661+0000 7f31c7f94640 1 -- 172.21.15.112:0/1963127424 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f31ac06de30 msgr2=0x7f31ac0702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:49.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.661+0000 7f31c7f94640 1 --2- 172.21.15.112:0/1963127424 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f31ac06de30 0x7f31ac0702f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f31c006ab70 tx=0x7f31b0006c90 comp rx=0 tx=0).stop 2024-09-10T03:40:49.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.661+0000 7f31c7f94640 1 -- 172.21.15.112:0/1963127424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31c0100880 msgr2=0x7f31c006adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:49.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.661+0000 7f31c7f94640 1 --2- 172.21.15.112:0/1963127424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31c0100880 0x7f31c006adf0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f31bc006010 tx=0x7f31bc0048f0 comp rx=0 tx=0).stop 2024-09-10T03:40:49.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.661+0000 7f31c7f94640 1 -- 172.21.15.112:0/1963127424 shutdown_connections 2024-09-10T03:40:49.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.661+0000 7f31c7f94640 1 --2- 172.21.15.112:0/1963127424 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f31ac06de30 0x7f31ac0702f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:49.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.661+0000 7f31c7f94640 1 --2- 172.21.15.112:0/1963127424 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f31c0101a80 0x7f31c019c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:49.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.661+0000 7f31c7f94640 1 --2- 172.21.15.112:0/1963127424 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31c0100880 0x7f31c006adf0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:49.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.661+0000 7f31c7f94640 1 -- 172.21.15.112:0/1963127424 >> 172.21.15.112:0/1963127424 conn(0x7f31c00fc010 msgr2=0x7f31c0104f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:49.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.661+0000 7f31c7f94640 1 -- 172.21.15.112:0/1963127424 shutdown_connections 2024-09-10T03:40:49.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:49.661+0000 7f31c7f94640 1 -- 172.21.15.112:0/1963127424 wait complete. 2024-09-10T03:40:50.219 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:49 smithi112 ceph-mon[27737]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 6.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-10T03:40:50.219 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:49 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:50.219 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:49 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:40:50.246 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -e sha1=3a5e3f16720f432aea965c40a8f6eb480095ec6c -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-10T03:40:50.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:49 smithi169 ceph-mon[35668]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 6.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-09-10T03:40:50.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:49 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:40:50.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:49 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:40:50.496 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:50.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.920+0000 7f3ea18e3640 1 -- 172.21.15.112:0/2565012461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e9c1023d0 msgr2=0x7f3e9c1047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:50.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.920+0000 7f3ea18e3640 1 --2- 172.21.15.112:0/2565012461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e9c1023d0 0x7f3e9c1047c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009c50 tx=0x7f3e8c02a9a0 comp rx=0 tx=0).stop 2024-09-10T03:40:50.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.920+0000 7f3ea18e3640 1 -- 172.21.15.112:0/2565012461 shutdown_connections 2024-09-10T03:40:50.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.920+0000 7f3ea18e3640 1 --2- 172.21.15.112:0/2565012461 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e9c104d00 0x7f3e9c1070f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:50.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.920+0000 7f3ea18e3640 1 --2- 172.21.15.112:0/2565012461 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e9c1023d0 0x7f3e9c1047c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:50.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.920+0000 7f3ea18e3640 1 -- 172.21.15.112:0/2565012461 >> 172.21.15.112:0/2565012461 conn(0x7f3e9c0fbff0 msgr2=0x7f3e9c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:50.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.920+0000 7f3ea18e3640 1 -- 172.21.15.112:0/2565012461 shutdown_connections 2024-09-10T03:40:50.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.920+0000 7f3ea18e3640 1 -- 172.21.15.112:0/2565012461 wait complete. 2024-09-10T03:40:50.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.921+0000 7f3ea18e3640 1 Processor -- start 2024-09-10T03:40:50.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.921+0000 7f3ea18e3640 1 -- start start 2024-09-10T03:40:50.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.921+0000 7f3ea18e3640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e9c1023d0 0x7f3e9c1975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:50.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.921+0000 7f3ea18e3640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e9c104d00 0x7f3e9c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:50.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.921+0000 7f3ea18e3640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c10a810 con 0x7f3e9c104d00 2024-09-10T03:40:50.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.921+0000 7f3ea18e3640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c10a690 con 0x7f3e9c1023d0 2024-09-10T03:40:50.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.922+0000 7f3e9a7fc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e9c104d00 0x7f3e9c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:50.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.922+0000 7f3e9a7fc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e9c104d00 0x7f3e9c197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:34064/0 (socket says 172.21.15.112:34064) 2024-09-10T03:40:50.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.922+0000 7f3e9a7fc640 1 -- 172.21.15.112:0/3071698093 learned_addr learned my addr 172.21.15.112:0/3071698093 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:50.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.922+0000 7f3e9affd640 1 --2- 172.21.15.112:0/3071698093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e9c1023d0 0x7f3e9c1975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:50.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.922+0000 7f3e9a7fc640 1 -- 172.21.15.112:0/3071698093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e9c1023d0 msgr2=0x7f3e9c1975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:50.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.922+0000 7f3e9a7fc640 1 --2- 172.21.15.112:0/3071698093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e9c1023d0 0x7f3e9c1975c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:50.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.922+0000 7f3e9a7fc640 1 -- 172.21.15.112:0/3071698093 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9c198180 con 0x7f3e9c104d00 2024-09-10T03:40:50.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.922+0000 7f3e9a7fc640 1 --2- 172.21.15.112:0/3071698093 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e9c104d00 0x7f3e9c197b00 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3e9000b3d0 tx=0x7f3e9000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:50.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.922+0000 7f3ea08e1640 1 -- 172.21.15.112:0/3071698093 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3e90015020 con 0x7f3e9c104d00 2024-09-10T03:40:50.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.922+0000 7f3ea08e1640 1 -- 172.21.15.112:0/3071698093 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3e90011af0 con 0x7f3e9c104d00 2024-09-10T03:40:50.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.922+0000 7f3ea18e3640 1 -- 172.21.15.112:0/3071698093 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e9c19cbc0 con 0x7f3e9c104d00 2024-09-10T03:40:50.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.922+0000 7f3ea08e1640 1 -- 172.21.15.112:0/3071698093 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3e90010d30 con 0x7f3e9c104d00 2024-09-10T03:40:50.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.922+0000 7f3ea18e3640 1 -- 172.21.15.112:0/3071698093 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e9c19d1d0 con 0x7f3e9c104d00 2024-09-10T03:40:50.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.923+0000 7f3ea18e3640 1 -- 172.21.15.112:0/3071698093 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e9c102830 con 0x7f3e9c104d00 2024-09-10T03:40:50.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.924+0000 7f3ea08e1640 1 -- 172.21.15.112:0/3071698093 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f3e90018020 con 0x7f3e9c104d00 2024-09-10T03:40:50.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.925+0000 7f3ea08e1640 1 --2- 172.21.15.112:0/3071698093 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3e8406de30 0x7f3e840702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:50.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.925+0000 7f3e9affd640 1 --2- 172.21.15.112:0/3071698093 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3e8406de30 0x7f3e840702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:50.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.925+0000 7f3ea08e1640 1 -- 172.21.15.112:0/3071698093 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7f3e9008de60 con 0x7f3e9c104d00 2024-09-10T03:40:50.927 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.925+0000 7f3e9affd640 1 --2- 172.21.15.112:0/3071698093 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3e8406de30 0x7f3e840702f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c02f860 tx=0x7f3e8c009360 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:50.930 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:50.928+0000 7f3ea08e1640 1 -- 172.21.15.112:0/3071698093 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3e9001f020 con 0x7f3e9c104d00 2024-09-10T03:40:51.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:51.094+0000 7f3ea18e3640 1 -- 172.21.15.112:0/3071698093 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) -- 0x7f3e9c19db40 con 0x7f3e9c104d00 2024-09-10T03:40:51.096 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:51.094+0000 7f3ea08e1640 1 -- 172.21.15.112:0/3071698093 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v43)=0 v43) ==== 135+0+0 (secure 0 0 0) 0x7f3e9005b980 con 0x7f3e9c104d00 2024-09-10T03:40:51.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:51.096+0000 7f3ea18e3640 1 -- 172.21.15.112:0/3071698093 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3e8406de30 msgr2=0x7f3e840702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:51.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:51.096+0000 7f3ea18e3640 1 --2- 172.21.15.112:0/3071698093 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3e8406de30 0x7f3e840702f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c02f860 tx=0x7f3e8c009360 comp rx=0 tx=0).stop 2024-09-10T03:40:51.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:51.096+0000 7f3ea18e3640 1 -- 172.21.15.112:0/3071698093 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e9c104d00 msgr2=0x7f3e9c197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:51.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:51.096+0000 7f3ea18e3640 1 --2- 172.21.15.112:0/3071698093 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e9c104d00 0x7f3e9c197b00 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3e9000b3d0 tx=0x7f3e9000b890 comp rx=0 tx=0).stop 2024-09-10T03:40:51.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:51.096+0000 7f3ea18e3640 1 -- 172.21.15.112:0/3071698093 shutdown_connections 2024-09-10T03:40:51.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:51.096+0000 7f3ea18e3640 1 --2- 172.21.15.112:0/3071698093 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f3e8406de30 0x7f3e840702f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:51.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:51.096+0000 7f3ea18e3640 1 --2- 172.21.15.112:0/3071698093 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e9c104d00 0x7f3e9c197b00 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:51.098 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:51.096+0000 7f3ea18e3640 1 --2- 172.21.15.112:0/3071698093 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3e9c1023d0 0x7f3e9c1975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:51.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:51.096+0000 7f3ea18e3640 1 -- 172.21.15.112:0/3071698093 >> 172.21.15.112:0/3071698093 conn(0x7f3e9c0fbff0 msgr2=0x7f3e9c0fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:51.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:51.096+0000 7f3ea18e3640 1 -- 172.21.15.112:0/3071698093 shutdown_connections 2024-09-10T03:40:51.099 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:51.096+0000 7f3ea18e3640 1 -- 172.21.15.112:0/3071698093 wait complete. 2024-09-10T03:40:51.699 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -e sha1=3a5e3f16720f432aea965c40a8f6eb480095ec6c -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-09-10T03:40:51.948 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:40:52.213 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:51 smithi112 ceph-mon[27737]: pgmap v191: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 5.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-09-10T03:40:52.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:51 smithi169 ceph-mon[35668]: pgmap v191: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 5.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-09-10T03:40:52.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.428+0000 7ff49e9be640 1 -- 172.21.15.112:0/4050562642 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff498102b80 msgr2=0x7ff498103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:52.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.428+0000 7ff49e9be640 1 --2- 172.21.15.112:0/4050562642 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff498102b80 0x7ff498103000 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff480009880 tx=0x7ff48002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:40:52.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.428+0000 7ff49e9be640 1 -- 172.21.15.112:0/4050562642 shutdown_connections 2024-09-10T03:40:52.439 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.428+0000 7ff49e9be640 1 --2- 172.21.15.112:0/4050562642 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff498102b80 0x7ff498103000 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:52.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.428+0000 7ff49e9be640 1 --2- 172.21.15.112:0/4050562642 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff498101980 0x7ff498101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:52.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.428+0000 7ff49e9be640 1 -- 172.21.15.112:0/4050562642 >> 172.21.15.112:0/4050562642 conn(0x7ff4980fd110 msgr2=0x7ff4980ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:40:52.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.428+0000 7ff49e9be640 1 -- 172.21.15.112:0/4050562642 shutdown_connections 2024-09-10T03:40:52.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.428+0000 7ff49e9be640 1 -- 172.21.15.112:0/4050562642 wait complete. 2024-09-10T03:40:52.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.428+0000 7ff49e9be640 1 Processor -- start 2024-09-10T03:40:52.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.428+0000 7ff49e9be640 1 -- start start 2024-09-10T03:40:52.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.429+0000 7ff49e9be640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff498101980 0x7ff498197490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:52.440 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.429+0000 7ff49e9be640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff498102b80 0x7ff4981979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:52.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.429+0000 7ff49e9be640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff49810a7a0 con 0x7ff498101980 2024-09-10T03:40:52.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.429+0000 7ff49e9be640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7ff49810a620 con 0x7ff498102b80 2024-09-10T03:40:52.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.429+0000 7ff497fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff498101980 0x7ff498197490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:52.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.429+0000 7ff497fff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff498101980 0x7ff498197490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:34080/0 (socket says 172.21.15.112:34080) 2024-09-10T03:40:52.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.429+0000 7ff497fff640 1 -- 172.21.15.112:0/2436819387 learned_addr learned my addr 172.21.15.112:0/2436819387 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:40:52.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.429+0000 7ff48ffff640 1 --2- 172.21.15.112:0/2436819387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff498102b80 0x7ff4981979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:52.441 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.429+0000 7ff497fff640 1 -- 172.21.15.112:0/2436819387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff498102b80 msgr2=0x7ff4981979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:40:52.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.429+0000 7ff497fff640 1 --2- 172.21.15.112:0/2436819387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff498102b80 0x7ff4981979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:40:52.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.429+0000 7ff497fff640 1 -- 172.21.15.112:0/2436819387 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff498197fc0 con 0x7ff498101980 2024-09-10T03:40:52.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.429+0000 7ff48ffff640 1 --2- 172.21.15.112:0/2436819387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff498102b80 0x7ff4981979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T03:40:52.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.429+0000 7ff497fff640 1 --2- 172.21.15.112:0/2436819387 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff498101980 0x7ff498197490 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff48800c8f0 tx=0x7ff48800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:52.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.429+0000 7ff495ffb640 1 -- 172.21.15.112:0/2436819387 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff488004020 con 0x7ff498101980 2024-09-10T03:40:52.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.429+0000 7ff495ffb640 1 -- 172.21.15.112:0/2436819387 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff488007e10 con 0x7ff498101980 2024-09-10T03:40:52.442 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.430+0000 7ff49e9be640 1 -- 172.21.15.112:0/2436819387 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff49819ca30 con 0x7ff498101980 2024-09-10T03:40:52.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.430+0000 7ff495ffb640 1 -- 172.21.15.112:0/2436819387 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff488003e10 con 0x7ff498101980 2024-09-10T03:40:52.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.430+0000 7ff49e9be640 1 -- 172.21.15.112:0/2436819387 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff49819cf90 con 0x7ff498101980 2024-09-10T03:40:52.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.431+0000 7ff49e9be640 1 -- 172.21.15.112:0/2436819387 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff49810af80 con 0x7ff498101980 2024-09-10T03:40:52.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.432+0000 7ff495ffb640 1 -- 172.21.15.112:0/2436819387 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7ff4880041c0 con 0x7ff498101980 2024-09-10T03:40:52.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.432+0000 7ff495ffb640 1 --2- 172.21.15.112:0/2436819387 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff48406de30 0x7ff4840702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:40:52.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.432+0000 7ff48ffff640 1 --2- 172.21.15.112:0/2436819387 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff48406de30 0x7ff4840702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:40:52.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.432+0000 7ff495ffb640 1 -- 172.21.15.112:0/2436819387 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7ff48808dca0 con 0x7ff498101980 2024-09-10T03:40:52.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.433+0000 7ff48ffff640 1 --2- 172.21.15.112:0/2436819387 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff48406de30 0x7ff4840702f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff4800096c0 tx=0x7ff480009650 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:40:52.443 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.437+0000 7ff495ffb640 1 -- 172.21.15.112:0/2436819387 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff488092070 con 0x7ff498101980 2024-09-10T03:40:52.620 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:40:52.619+0000 7ff49e9be640 1 -- 172.21.15.112:0/2436819387 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:3a5e3f16720f432aea965c40a8f6eb480095ec6c", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) -- 0x7ff498062630 con 0x7ff48406de30 2024-09-10T03:40:54.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:53 smithi169 ceph-mon[35668]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:3a5e3f16720f432aea965c40a8f6eb480095ec6c", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:40:54.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:53 smithi169 ceph-mon[35668]: pgmap v192: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 938 B/s wr, 1 op/s 2024-09-10T03:40:54.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:53 smithi112 ceph-mon[27737]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:3a5e3f16720f432aea965c40a8f6eb480095ec6c", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:40:54.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:53 smithi112 ceph-mon[27737]: pgmap v192: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 938 B/s wr, 1 op/s 2024-09-10T03:40:56.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:55 smithi169 ceph-mon[35668]: pgmap v193: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-09-10T03:40:56.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:55 smithi112 ceph-mon[27737]: pgmap v193: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 938 B/s wr, 0 op/s 2024-09-10T03:40:58.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:57 smithi169 ceph-mon[35668]: pgmap v194: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-09-10T03:40:58.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:57 smithi112 ceph-mon[27737]: pgmap v194: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-09-10T03:41:00.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:40:59 smithi169 ceph-mon[35668]: pgmap v195: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-09-10T03:41:00.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:40:59 smithi112 ceph-mon[27737]: pgmap v195: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-09-10T03:41:02.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:01 smithi169 ceph-mon[35668]: pgmap v196: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-09-10T03:41:02.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:01 smithi112 ceph-mon[27737]: pgmap v196: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-09-10T03:41:04.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:03 smithi169 ceph-mon[35668]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:04.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:03 smithi112 ceph-mon[27737]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:05.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:04 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:41:05.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:04 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:41:06.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:05 smithi169 ceph-mon[35668]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:06.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:05 smithi112 ceph-mon[27737]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:08.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:07 smithi169 ceph-mon[35668]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:08.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:07 smithi112 ceph-mon[27737]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:10.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:09 smithi169 ceph-mon[35668]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:10.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:09 smithi112 ceph-mon[27737]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:12.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:11 smithi169 ceph-mon[35668]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:12.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:11 smithi112 ceph-mon[27737]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:14.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:13 smithi169 ceph-mon[35668]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:14.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:13 smithi112 ceph-mon[27737]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:16.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:15 smithi169 ceph-mon[35668]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:16.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:15 smithi112 ceph-mon[27737]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:18.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:17 smithi169 ceph-mon[35668]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:18.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:17 smithi112 ceph-mon[27737]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:20.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:19 smithi169 ceph-mon[35668]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:20.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:19 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:41:20.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:19 smithi112 ceph-mon[27737]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:20.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:19 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:41:22.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:21 smithi169 ceph-mon[35668]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-09-10T03:41:22.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:21 smithi112 ceph-mon[27737]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-09-10T03:41:24.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:23 smithi169 ceph-mon[35668]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-09-10T03:41:24.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:23 smithi112 ceph-mon[27737]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-09-10T03:41:26.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:25 smithi169 ceph-mon[35668]: pgmap v208: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-09-10T03:41:26.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:25 smithi112 ceph-mon[27737]: pgmap v208: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-09-10T03:41:28.147 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:27 smithi112 ceph-mon[27737]: pgmap v209: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-09-10T03:41:28.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:27 smithi169 ceph-mon[35668]: pgmap v209: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-09-10T03:41:28.888 INFO:teuthology.orchestra.run.smithi112.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:3a5e3f16720f432aea965c40a8f6eb480095ec6c 2024-09-10T03:41:28.888 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:28.886+0000 7ff495ffb640 1 -- 172.21.15.112:0/2436819387 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+89 (secure 0 0 0) 0x7ff498062630 con 0x7ff48406de30 2024-09-10T03:41:28.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:28.889+0000 7ff49e9be640 1 -- 172.21.15.112:0/2436819387 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff48406de30 msgr2=0x7ff4840702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:28.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:28.889+0000 7ff49e9be640 1 --2- 172.21.15.112:0/2436819387 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff48406de30 0x7ff4840702f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff4800096c0 tx=0x7ff480009650 comp rx=0 tx=0).stop 2024-09-10T03:41:28.891 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:28.889+0000 7ff49e9be640 1 -- 172.21.15.112:0/2436819387 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff498101980 msgr2=0x7ff498197490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:28.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:28.889+0000 7ff49e9be640 1 --2- 172.21.15.112:0/2436819387 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff498101980 0x7ff498197490 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff48800c8f0 tx=0x7ff48800cdb0 comp rx=0 tx=0).stop 2024-09-10T03:41:28.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:28.890+0000 7ff49e9be640 1 -- 172.21.15.112:0/2436819387 shutdown_connections 2024-09-10T03:41:28.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:28.890+0000 7ff49e9be640 1 --2- 172.21.15.112:0/2436819387 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7ff48406de30 0x7ff4840702f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:28.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:28.890+0000 7ff49e9be640 1 --2- 172.21.15.112:0/2436819387 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff498102b80 0x7ff4981979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:28.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:28.890+0000 7ff49e9be640 1 --2- 172.21.15.112:0/2436819387 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff498101980 0x7ff498197490 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:28.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:28.890+0000 7ff49e9be640 1 -- 172.21.15.112:0/2436819387 >> 172.21.15.112:0/2436819387 conn(0x7ff4980fd110 msgr2=0x7ff498106030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:41:28.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:28.890+0000 7ff49e9be640 1 -- 172.21.15.112:0/2436819387 shutdown_connections 2024-09-10T03:41:28.892 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:28.890+0000 7ff49e9be640 1 -- 172.21.15.112:0/2436819387 wait complete. 2024-09-10T03:41:29.502 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -e sha1=3a5e3f16720f432aea965c40a8f6eb480095ec6c -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-09-10T03:41:29.885 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:41:30.251 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:29 smithi112 ceph-mon[27737]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:3a5e3f16720f432aea965c40a8f6eb480095ec6c 2024-09-10T03:41:30.251 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:29 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:41:30.251 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:29 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:41:30.252 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:29 smithi112 ceph-mon[27737]: pgmap v210: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-09-10T03:41:30.252 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:29 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:41:30.252 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:29 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:41:30.252 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:29 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:41:30.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:29 smithi169 ceph-mon[35668]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:3a5e3f16720f432aea965c40a8f6eb480095ec6c 2024-09-10T03:41:30.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:29 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:41:30.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:29 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:41:30.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:29 smithi169 ceph-mon[35668]: pgmap v210: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-09-10T03:41:30.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:29 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:41:30.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:29 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:41:30.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:29 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:41:30.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.591+0000 7f56a30dc640 1 -- 172.21.15.112:0/3029256258 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f569c100840 msgr2=0x7f569c100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:30.611 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.591+0000 7f56a30dc640 1 --2- 172.21.15.112:0/3029256258 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f569c100840 0x7f569c100c40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5690009880 tx=0x7f569002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:41:30.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.591+0000 7f56a30dc640 1 -- 172.21.15.112:0/3029256258 shutdown_connections 2024-09-10T03:41:30.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.591+0000 7f56a30dc640 1 --2- 172.21.15.112:0/3029256258 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f569c101a40 0x7f569c101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:30.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.591+0000 7f56a30dc640 1 --2- 172.21.15.112:0/3029256258 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f569c100840 0x7f569c100c40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:30.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.591+0000 7f56a30dc640 1 -- 172.21.15.112:0/3029256258 >> 172.21.15.112:0/3029256258 conn(0x7f569c0fbff0 msgr2=0x7f569c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:41:30.612 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.591+0000 7f56a30dc640 1 -- 172.21.15.112:0/3029256258 shutdown_connections 2024-09-10T03:41:30.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.591+0000 7f56a30dc640 1 -- 172.21.15.112:0/3029256258 wait complete. 2024-09-10T03:41:30.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.592+0000 7f56a30dc640 1 Processor -- start 2024-09-10T03:41:30.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.592+0000 7f56a30dc640 1 -- start start 2024-09-10T03:41:30.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.593+0000 7f56a30dc640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f569c100840 0x7f569c06a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:41:30.613 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.593+0000 7f56a30dc640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f569c101a40 0x7f569c19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:41:30.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.593+0000 7f56a30dc640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f569c109680 con 0x7f569c100840 2024-09-10T03:41:30.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.593+0000 7f56a30dc640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f569c109500 con 0x7f569c101a40 2024-09-10T03:41:30.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.593+0000 7f569bfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f569c101a40 0x7f569c19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:41:30.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.593+0000 7f569bfff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f569c101a40 0x7f569c19e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.112:58558/0 (socket says 172.21.15.112:58558) 2024-09-10T03:41:30.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.593+0000 7f569bfff640 1 -- 172.21.15.112:0/2703544881 learned_addr learned my addr 172.21.15.112:0/2703544881 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:41:30.614 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.593+0000 7f56a0e78640 1 --2- 172.21.15.112:0/2703544881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f569c100840 0x7f569c06a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:41:30.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.594+0000 7f569bfff640 1 -- 172.21.15.112:0/2703544881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f569c100840 msgr2=0x7f569c06a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:30.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.594+0000 7f569bfff640 1 --2- 172.21.15.112:0/2703544881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f569c100840 0x7f569c06a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:30.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.594+0000 7f569bfff640 1 -- 172.21.15.112:0/2703544881 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f569c19eb60 con 0x7f569c101a40 2024-09-10T03:41:30.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.594+0000 7f56a0e78640 1 --2- 172.21.15.112:0/2703544881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f569c100840 0x7f569c06a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T03:41:30.615 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.594+0000 7f569bfff640 1 --2- 172.21.15.112:0/2703544881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f569c101a40 0x7f569c19e620 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f568800e9a0 tx=0x7f568800eda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:41:30.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.594+0000 7f5699ffb640 1 -- 172.21.15.112:0/2703544881 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f568800cb80 con 0x7f569c101a40 2024-09-10T03:41:30.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.594+0000 7f56a30dc640 1 -- 172.21.15.112:0/2703544881 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f569c19ed30 con 0x7f569c101a40 2024-09-10T03:41:30.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.594+0000 7f5699ffb640 1 -- 172.21.15.112:0/2703544881 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f56880054e0 con 0x7f569c101a40 2024-09-10T03:41:30.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.595+0000 7f5699ffb640 1 -- 172.21.15.112:0/2703544881 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f5688010430 con 0x7f569c101a40 2024-09-10T03:41:30.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.595+0000 7f56a30dc640 1 -- 172.21.15.112:0/2703544881 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f569c19f260 con 0x7f569c101a40 2024-09-10T03:41:30.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.597+0000 7f56a30dc640 1 -- 172.21.15.112:0/2703544881 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f569c100c40 con 0x7f569c101a40 2024-09-10T03:41:30.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.597+0000 7f5699ffb640 1 -- 172.21.15.112:0/2703544881 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f568800cd20 con 0x7f569c101a40 2024-09-10T03:41:30.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.597+0000 7f5699ffb640 1 --2- 172.21.15.112:0/2703544881 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f568c06de80 0x7f568c070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:41:30.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.598+0000 7f56a0e78640 1 --2- 172.21.15.112:0/2703544881 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f568c06de80 0x7f568c070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:41:30.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.598+0000 7f5699ffb640 1 -- 172.21.15.112:0/2703544881 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7f568808dcf0 con 0x7f569c101a40 2024-09-10T03:41:30.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.599+0000 7f56a0e78640 1 --2- 172.21.15.112:0/2703544881 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f568c06de80 0x7f568c070340 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5690000c00 tx=0x7f5690005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:41:30.617 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.606+0000 7f5699ffb640 1 -- 172.21.15.112:0/2703544881 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f568805b820 con 0x7f569c101a40 2024-09-10T03:41:30.893 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:30 smithi112 ceph-mon[27737]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:3a5e3f16720f432aea965c40a8f6eb480095ec6c 2024-09-10T03:41:30.988 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.987+0000 7f56a30dc640 1 -- 172.21.15.112:0/2703544881 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) -- 0x7f569c062630 con 0x7f568c06de80 2024-09-10T03:41:30.990 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.989+0000 7f5699ffb640 1 -- 172.21.15.112:0/2703544881 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7f569c062630 con 0x7f568c06de80 2024-09-10T03:41:30.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.992+0000 7f56a30dc640 1 -- 172.21.15.112:0/2703544881 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f568c06de80 msgr2=0x7f568c070340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:30.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.992+0000 7f56a30dc640 1 --2- 172.21.15.112:0/2703544881 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f568c06de80 0x7f568c070340 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5690000c00 tx=0x7f5690005a20 comp rx=0 tx=0).stop 2024-09-10T03:41:30.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.992+0000 7f56a30dc640 1 -- 172.21.15.112:0/2703544881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f569c101a40 msgr2=0x7f569c19e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:30.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.992+0000 7f56a30dc640 1 --2- 172.21.15.112:0/2703544881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f569c101a40 0x7f569c19e620 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f568800e9a0 tx=0x7f568800eda0 comp rx=0 tx=0).stop 2024-09-10T03:41:30.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.992+0000 7f56a30dc640 1 -- 172.21.15.112:0/2703544881 shutdown_connections 2024-09-10T03:41:30.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.992+0000 7f56a30dc640 1 --2- 172.21.15.112:0/2703544881 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f568c06de80 0x7f568c070340 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:30.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.992+0000 7f56a30dc640 1 --2- 172.21.15.112:0/2703544881 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f569c101a40 0x7f569c19e620 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:30.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.992+0000 7f56a30dc640 1 --2- 172.21.15.112:0/2703544881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f569c100840 0x7f569c06a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:30.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.992+0000 7f56a30dc640 1 -- 172.21.15.112:0/2703544881 >> 172.21.15.112:0/2703544881 conn(0x7f569c0fbff0 msgr2=0x7f569c104ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:41:30.993 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.992+0000 7f56a30dc640 1 -- 172.21.15.112:0/2703544881 shutdown_connections 2024-09-10T03:41:30.994 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:30.992+0000 7f56a30dc640 1 -- 172.21.15.112:0/2703544881 wait complete. 2024-09-10T03:41:31.005 INFO:teuthology.orchestra.run.smithi112.stdout:true 2024-09-10T03:41:31.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.124+0000 7f7be7a76640 1 -- 172.21.15.112:0/525719127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7be00ffc00 msgr2=0x7f7be0100000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:31.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.124+0000 7f7be7a76640 1 --2- 172.21.15.112:0/525719127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7be00ffc00 0x7f7be0100000 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc009880 tx=0x7f7bdc02a4b0 comp rx=0 tx=0).stop 2024-09-10T03:41:31.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.125+0000 7f7be7a76640 1 -- 172.21.15.112:0/525719127 shutdown_connections 2024-09-10T03:41:31.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.125+0000 7f7be7a76640 1 --2- 172.21.15.112:0/525719127 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7be0102be0 0x7f7be0103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:31.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.125+0000 7f7be7a76640 1 --2- 172.21.15.112:0/525719127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7be00ffc00 0x7f7be0100000 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:31.126 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.125+0000 7f7be7a76640 1 -- 172.21.15.112:0/525719127 >> 172.21.15.112:0/525719127 conn(0x7f7be00fb750 msgr2=0x7f7be00fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:41:31.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.125+0000 7f7be7a76640 1 -- 172.21.15.112:0/525719127 shutdown_connections 2024-09-10T03:41:31.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.125+0000 7f7be7a76640 1 -- 172.21.15.112:0/525719127 wait complete. 2024-09-10T03:41:31.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.126+0000 7f7be7a76640 1 Processor -- start 2024-09-10T03:41:31.127 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.126+0000 7f7be7a76640 1 -- start start 2024-09-10T03:41:31.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.127+0000 7f7be7a76640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7be00ffc00 0x7f7be0197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:41:31.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.127+0000 7f7be7a76640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7be0102be0 0x7f7be01979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:41:31.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.127+0000 7f7be7a76640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7be010a760 con 0x7f7be00ffc00 2024-09-10T03:41:31.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.127+0000 7f7be7a76640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f7be010a5e0 con 0x7f7be0102be0 2024-09-10T03:41:31.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.127+0000 7f7be5812640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7be00ffc00 0x7f7be0197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:41:31.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.127+0000 7f7be5812640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7be00ffc00 0x7f7be0197480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:33278/0 (socket says 172.21.15.112:33278) 2024-09-10T03:41:31.129 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.127+0000 7f7be5812640 1 -- 172.21.15.112:0/1256967869 learned_addr learned my addr 172.21.15.112:0/1256967869 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:41:31.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.127+0000 7f7be5011640 1 --2- 172.21.15.112:0/1256967869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7be0102be0 0x7f7be01979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:41:31.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.128+0000 7f7be5812640 1 -- 172.21.15.112:0/1256967869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7be0102be0 msgr2=0x7f7be01979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:31.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.128+0000 7f7be5812640 1 --2- 172.21.15.112:0/1256967869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7be0102be0 0x7f7be01979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:31.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.128+0000 7f7be5812640 1 -- 172.21.15.112:0/1256967869 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be0198040 con 0x7f7be00ffc00 2024-09-10T03:41:31.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.129+0000 7f7be5812640 1 --2- 172.21.15.112:0/1256967869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7be00ffc00 0x7f7be0197480 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc02aaa0 tx=0x7f7bdc0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:41:31.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.129+0000 7f7bd6ffd640 1 -- 172.21.15.112:0/1256967869 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f7bdc03f070 con 0x7f7be00ffc00 2024-09-10T03:41:31.130 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.129+0000 7f7bd6ffd640 1 -- 172.21.15.112:0/1256967869 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7bdc004a60 con 0x7f7be00ffc00 2024-09-10T03:41:31.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.129+0000 7f7bd6ffd640 1 -- 172.21.15.112:0/1256967869 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f7bdc035030 con 0x7f7be00ffc00 2024-09-10T03:41:31.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.130+0000 7f7be7a76640 1 -- 172.21.15.112:0/1256967869 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7be019ca20 con 0x7f7be00ffc00 2024-09-10T03:41:31.131 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.130+0000 7f7be7a76640 1 -- 172.21.15.112:0/1256967869 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7be019cfb0 con 0x7f7be00ffc00 2024-09-10T03:41:31.133 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.131+0000 7f7be7a76640 1 -- 172.21.15.112:0/1256967869 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7be0100000 con 0x7f7be00ffc00 2024-09-10T03:41:31.141 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.141+0000 7f7bd6ffd640 1 -- 172.21.15.112:0/1256967869 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f7bdc042070 con 0x7f7be00ffc00 2024-09-10T03:41:31.142 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.141+0000 7f7bd6ffd640 1 --2- 172.21.15.112:0/1256967869 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f7bcc06df20 0x7f7bcc0703e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:41:31.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.142+0000 7f7bd6ffd640 1 -- 172.21.15.112:0/1256967869 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7f7bdc0ae660 con 0x7f7be00ffc00 2024-09-10T03:41:31.143 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.142+0000 7f7be5011640 1 --2- 172.21.15.112:0/1256967869 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f7bcc06df20 0x7f7bcc0703e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:41:31.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.144+0000 7f7bd6ffd640 1 -- 172.21.15.112:0/1256967869 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7bdc049030 con 0x7f7be00ffc00 2024-09-10T03:41:31.145 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.144+0000 7f7be5011640 1 --2- 172.21.15.112:0/1256967869 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f7bcc06df20 0x7f7bcc0703e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7bd00095f0 tx=0x7f7bd0005f90 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:41:31.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:30 smithi169 ceph-mon[35668]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:3a5e3f16720f432aea965c40a8f6eb480095ec6c 2024-09-10T03:41:31.523 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.522+0000 7f7be7a76640 1 -- 172.21.15.112:0/1256967869 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) -- 0x7f7be0062740 con 0x7f7bcc06df20 2024-09-10T03:41:31.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.524+0000 7f7bd6ffd640 1 -- 172.21.15.112:0/1256967869 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7f7be0062740 con 0x7f7bcc06df20 2024-09-10T03:41:31.527 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.526+0000 7f7be7a76640 1 -- 172.21.15.112:0/1256967869 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f7bcc06df20 msgr2=0x7f7bcc0703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:31.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.526+0000 7f7be7a76640 1 --2- 172.21.15.112:0/1256967869 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f7bcc06df20 0x7f7bcc0703e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7bd00095f0 tx=0x7f7bd0005f90 comp rx=0 tx=0).stop 2024-09-10T03:41:31.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.527+0000 7f7be7a76640 1 -- 172.21.15.112:0/1256967869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7be00ffc00 msgr2=0x7f7be0197480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:31.528 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.527+0000 7f7be7a76640 1 --2- 172.21.15.112:0/1256967869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7be00ffc00 0x7f7be0197480 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc02aaa0 tx=0x7f7bdc0048f0 comp rx=0 tx=0).stop 2024-09-10T03:41:31.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.527+0000 7f7be7a76640 1 -- 172.21.15.112:0/1256967869 shutdown_connections 2024-09-10T03:41:31.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.527+0000 7f7be7a76640 1 --2- 172.21.15.112:0/1256967869 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f7bcc06df20 0x7f7bcc0703e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:31.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.527+0000 7f7be7a76640 1 --2- 172.21.15.112:0/1256967869 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7be0102be0 0x7f7be01979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:31.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.527+0000 7f7be7a76640 1 --2- 172.21.15.112:0/1256967869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7be00ffc00 0x7f7be0197480 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:31.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.527+0000 7f7be7a76640 1 -- 172.21.15.112:0/1256967869 >> 172.21.15.112:0/1256967869 conn(0x7f7be00fb750 msgr2=0x7f7be0106090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:41:31.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.527+0000 7f7be7a76640 1 -- 172.21.15.112:0/1256967869 shutdown_connections 2024-09-10T03:41:31.529 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.527+0000 7f7be7a76640 1 -- 172.21.15.112:0/1256967869 wait complete. 2024-09-10T03:41:31.661 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.659+0000 7f2eff7c7640 1 -- 172.21.15.112:0/1443734749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ef8100840 msgr2=0x7f2ef8100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:31.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.659+0000 7f2eff7c7640 1 --2- 172.21.15.112:0/1443734749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ef8100840 0x7f2ef8100c40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009880 tx=0x7f2ef402a4a0 comp rx=0 tx=0).stop 2024-09-10T03:41:31.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.659+0000 7f2eff7c7640 1 -- 172.21.15.112:0/1443734749 shutdown_connections 2024-09-10T03:41:31.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.659+0000 7f2eff7c7640 1 --2- 172.21.15.112:0/1443734749 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ef8101a40 0x7f2ef8101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:31.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.659+0000 7f2eff7c7640 1 --2- 172.21.15.112:0/1443734749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ef8100840 0x7f2ef8100c40 secure :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009880 tx=0x7f2ef402a4a0 comp rx=0 tx=0).stop 2024-09-10T03:41:31.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.659+0000 7f2eff7c7640 1 -- 172.21.15.112:0/1443734749 >> 172.21.15.112:0/1443734749 conn(0x7f2ef80fbff0 msgr2=0x7f2ef80fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:41:31.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.660+0000 7f2eff7c7640 1 -- 172.21.15.112:0/1443734749 shutdown_connections 2024-09-10T03:41:31.662 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.660+0000 7f2eff7c7640 1 -- 172.21.15.112:0/1443734749 wait complete. 2024-09-10T03:41:31.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.661+0000 7f2eff7c7640 1 Processor -- start 2024-09-10T03:41:31.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.661+0000 7f2eff7c7640 1 -- start start 2024-09-10T03:41:31.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.662+0000 7f2eff7c7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ef8101a40 0x7f2ef8195630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:41:31.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.662+0000 7f2eff7c7640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ef8195b70 0x7f2ef819ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:41:31.663 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.662+0000 7f2eff7c7640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef8108570 con 0x7f2ef8195b70 2024-09-10T03:41:31.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.662+0000 7f2eff7c7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef81083f0 con 0x7f2ef8101a40 2024-09-10T03:41:31.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.662+0000 7f2efd563640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ef8101a40 0x7f2ef8195630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:41:31.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.662+0000 7f2efd563640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ef8101a40 0x7f2ef8195630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.112:58574/0 (socket says 172.21.15.112:58574) 2024-09-10T03:41:31.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.662+0000 7f2efd563640 1 -- 172.21.15.112:0/3540510103 learned_addr learned my addr 172.21.15.112:0/3540510103 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:41:31.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.662+0000 7f2efcd62640 1 --2- 172.21.15.112:0/3540510103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ef8195b70 0x7f2ef819ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:41:31.664 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.662+0000 7f2efd563640 1 -- 172.21.15.112:0/3540510103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ef8195b70 msgr2=0x7f2ef819ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:31.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.664+0000 7f2efd563640 1 --2- 172.21.15.112:0/3540510103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ef8195b70 0x7f2ef819ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:31.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.664+0000 7f2efd563640 1 -- 172.21.15.112:0/3540510103 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ef81960a0 con 0x7f2ef8101a40 2024-09-10T03:41:31.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.664+0000 7f2efcd62640 1 --2- 172.21.15.112:0/3540510103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ef8195b70 0x7f2ef819ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T03:41:31.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.664+0000 7f2efd563640 1 --2- 172.21.15.112:0/3540510103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ef8101a40 0x7f2ef8195630 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2ef81018a0 tx=0x7f2ef4004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:41:31.665 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.665+0000 7f2eee7fc640 1 -- 172.21.15.112:0/3540510103 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2ef4040070 con 0x7f2ef8101a40 2024-09-10T03:41:31.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.665+0000 7f2eee7fc640 1 -- 172.21.15.112:0/3540510103 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2ef4004950 con 0x7f2ef8101a40 2024-09-10T03:41:31.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.665+0000 7f2eff7c7640 1 -- 172.21.15.112:0/3540510103 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ef819b0b0 con 0x7f2ef8101a40 2024-09-10T03:41:31.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.665+0000 7f2eee7fc640 1 -- 172.21.15.112:0/3540510103 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2ef4035020 con 0x7f2ef8101a40 2024-09-10T03:41:31.667 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.665+0000 7f2eff7c7640 1 -- 172.21.15.112:0/3540510103 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ef819b500 con 0x7f2ef8101a40 2024-09-10T03:41:31.668 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.667+0000 7f2eff7c7640 1 -- 172.21.15.112:0/3540510103 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2edc0052c0 con 0x7f2ef8101a40 2024-09-10T03:41:31.669 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.668+0000 7f2eee7fc640 1 -- 172.21.15.112:0/3540510103 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f2ef4043070 con 0x7f2ef8101a40 2024-09-10T03:41:31.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.669+0000 7f2eee7fc640 1 --2- 172.21.15.112:0/3540510103 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2ee4070090 0x7f2ee4072550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:41:31.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.669+0000 7f2efcd62640 1 --2- 172.21.15.112:0/3540510103 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2ee4070090 0x7f2ee4072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:41:31.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.670+0000 7f2eee7fc640 1 -- 172.21.15.112:0/3540510103 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7f2ef407a1a0 con 0x7f2ef8101a40 2024-09-10T03:41:31.671 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.670+0000 7f2efcd62640 1 --2- 172.21.15.112:0/3540510103 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2ee4070090 0x7f2ee4072550 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8196ae0 tx=0x7f2ee8005f90 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:41:31.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:31.676+0000 7f2eee7fc640 1 -- 172.21.15.112:0/3540510103 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2ef40804b0 con 0x7f2ef8101a40 2024-09-10T03:41:31.937 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:31 smithi112 ceph-mon[27737]: pgmap v211: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-09-10T03:41:31.937 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:31 smithi112 ceph-mon[27737]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:41:32.044 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.043+0000 7f2eff7c7640 1 -- 172.21.15.112:0/3540510103 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f2edc000bf0 con 0x7f2ee4070090 2024-09-10T03:41:32.063 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.062+0000 7f2eee7fc640 1 -- 172.21.15.112:0/3540510103 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3792 (secure 0 0 0) 0x7f2edc000bf0 con 0x7f2ee4070090 2024-09-10T03:41:32.064 INFO:teuthology.orchestra.run.smithi112.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-10T03:41:32.064 INFO:teuthology.orchestra.run.smithi112.stdout:alertmanager.smithi112 smithi112 *:9093,9094 running (4m) 45s ago 6m 26.1M - 0.25.0 c8568f914cd2 28468ac02dbb 2024-09-10T03:41:32.064 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-exporter.smithi112 smithi112 running (6m) 45s ago 6m 8208k - 19.1.1-298-gb53b6d3e 54467abc55f8 0ff34f471a47 2024-09-10T03:41:32.064 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-exporter.smithi169 smithi169 running (5m) 49s ago 5m 5263k - 19.1.1-298-gb53b6d3e 54467abc55f8 f430c6826280 2024-09-10T03:41:32.064 INFO:teuthology.orchestra.run.smithi112.stdout:crash.smithi112 smithi112 running (6m) 45s ago 6m 6899k - 19.1.1-298-gb53b6d3e 54467abc55f8 f7d1df4e46c8 2024-09-10T03:41:32.064 INFO:teuthology.orchestra.run.smithi112.stdout:crash.smithi169 smithi169 running (5m) 49s ago 5m 6899k - 19.1.1-298-gb53b6d3e 54467abc55f8 a27310b2722f 2024-09-10T03:41:32.064 INFO:teuthology.orchestra.run.smithi112.stdout:grafana.smithi112 smithi112 *:3000 running (4m) 45s ago 6m 80.6M - 9.4.12 2bacad6d85d8 444b4b1d4b44 2024-09-10T03:41:32.064 INFO:teuthology.orchestra.run.smithi112.stdout:mds.cephfs.smithi112.ifnhnl smithi112 running (69s) 45s ago 68s 11.7M - 19.1.1-298-gb53b6d3e 54467abc55f8 5737c46b4900 2024-09-10T03:41:32.064 INFO:teuthology.orchestra.run.smithi112.stdout:mds.cephfs.smithi112.uuenbo smithi112 running (62s) 45s ago 62s 16.4M - 19.1.1-298-gb53b6d3e 54467abc55f8 d62acd5a079a 2024-09-10T03:41:32.064 INFO:teuthology.orchestra.run.smithi112.stdout:mds.cephfs.smithi169.gcmhul smithi169 running (60s) 49s ago 59s 13.1M - 19.1.1-298-gb53b6d3e 54467abc55f8 36b459a6aef7 2024-09-10T03:41:32.064 INFO:teuthology.orchestra.run.smithi112.stdout:mds.cephfs.smithi169.wfmonc smithi169 running (66s) 49s ago 66s 14.7M - 19.1.1-298-gb53b6d3e 54467abc55f8 13b086adcdc0 2024-09-10T03:41:32.064 INFO:teuthology.orchestra.run.smithi112.stdout:mgr.smithi112.pqrlrb smithi112 *:9283,8765,8443 running (7m) 45s ago 7m 499M - 19.1.1-298-gb53b6d3e 54467abc55f8 5bcbeef123a8 2024-09-10T03:41:32.064 INFO:teuthology.orchestra.run.smithi112.stdout:mgr.smithi169.bkwgjl smithi169 *:8443,9283,8765 running (5m) 49s ago 5m 441M - 19.1.1-298-gb53b6d3e 54467abc55f8 b78844ecd2af 2024-09-10T03:41:32.065 INFO:teuthology.orchestra.run.smithi112.stdout:mon.smithi112 smithi112 running (7m) 45s ago 8m 47.4M 2048M 19.1.1-298-gb53b6d3e 54467abc55f8 91597bdf09f7 2024-09-10T03:41:32.065 INFO:teuthology.orchestra.run.smithi112.stdout:mon.smithi169 smithi169 running (5m) 49s ago 5m 35.2M 2048M 19.1.1-298-gb53b6d3e 54467abc55f8 38476c275eb5 2024-09-10T03:41:32.065 INFO:teuthology.orchestra.run.smithi112.stdout:node-exporter.smithi112 smithi112 *:9100 running (6m) 45s ago 6m 15.8M - 1.5.0 0da6a335fe13 736b09a514a8 2024-09-10T03:41:32.065 INFO:teuthology.orchestra.run.smithi112.stdout:node-exporter.smithi169 smithi169 *:9100 running (5m) 49s ago 5m 14.8M - 1.5.0 0da6a335fe13 b6d2ee0493db 2024-09-10T03:41:32.065 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0 smithi112 running (4m) 45s ago 4m 56.2M 3946M 19.1.1-298-gb53b6d3e 54467abc55f8 274fcfdbe0d8 2024-09-10T03:41:32.065 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1 smithi112 running (3m) 45s ago 3m 56.6M 3946M 19.1.1-298-gb53b6d3e 54467abc55f8 c48f7eddf6a2 2024-09-10T03:41:32.065 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2 smithi112 running (3m) 45s ago 3m 48.5M 3946M 19.1.1-298-gb53b6d3e 54467abc55f8 2885646ffe21 2024-09-10T03:41:32.065 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3 smithi169 running (3m) 49s ago 3m 45.2M 4970M 19.1.1-298-gb53b6d3e 54467abc55f8 b55e9df427b0 2024-09-10T03:41:32.065 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4 smithi169 running (2m) 49s ago 2m 52.0M 4970M 19.1.1-298-gb53b6d3e 54467abc55f8 f3527032352f 2024-09-10T03:41:32.065 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5 smithi169 running (2m) 49s ago 2m 50.6M 4970M 19.1.1-298-gb53b6d3e 54467abc55f8 b4df34a99808 2024-09-10T03:41:32.065 INFO:teuthology.orchestra.run.smithi112.stdout:prometheus.smithi112 smithi112 *:9095 running (4m) 45s ago 6m 59.1M - 2.43.0 a07b618ecd1d 3cb9cb258756 2024-09-10T03:41:32.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.065+0000 7f2eff7c7640 1 -- 172.21.15.112:0/3540510103 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2ee4070090 msgr2=0x7f2ee4072550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:32.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.065+0000 7f2eff7c7640 1 --2- 172.21.15.112:0/3540510103 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2ee4070090 0x7f2ee4072550 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8196ae0 tx=0x7f2ee8005f90 comp rx=0 tx=0).stop 2024-09-10T03:41:32.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.065+0000 7f2eff7c7640 1 -- 172.21.15.112:0/3540510103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ef8101a40 msgr2=0x7f2ef8195630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:32.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.065+0000 7f2eff7c7640 1 --2- 172.21.15.112:0/3540510103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ef8101a40 0x7f2ef8195630 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2ef81018a0 tx=0x7f2ef4004600 comp rx=0 tx=0).stop 2024-09-10T03:41:32.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.066+0000 7f2eff7c7640 1 -- 172.21.15.112:0/3540510103 shutdown_connections 2024-09-10T03:41:32.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.066+0000 7f2eff7c7640 1 --2- 172.21.15.112:0/3540510103 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f2ee4070090 0x7f2ee4072550 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.066+0000 7f2eff7c7640 1 --2- 172.21.15.112:0/3540510103 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ef8195b70 0x7f2ef819ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.066+0000 7f2eff7c7640 1 --2- 172.21.15.112:0/3540510103 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2ef8101a40 0x7f2ef8195630 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.066+0000 7f2eff7c7640 1 -- 172.21.15.112:0/3540510103 >> 172.21.15.112:0/3540510103 conn(0x7f2ef80fbff0 msgr2=0x7f2ef8104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:41:32.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.066+0000 7f2eff7c7640 1 -- 172.21.15.112:0/3540510103 shutdown_connections 2024-09-10T03:41:32.067 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.066+0000 7f2eff7c7640 1 -- 172.21.15.112:0/3540510103 wait complete. 2024-09-10T03:41:32.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.154+0000 7fb4a1dcb640 1 -- 172.21.15.112:0/867346633 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb49c101a40 msgr2=0x7fb49c101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:32.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.154+0000 7fb4a1dcb640 1 --2- 172.21.15.112:0/867346633 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb49c101a40 0x7fb49c101ec0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb484009880 tx=0x7fb48402a4a0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.154+0000 7fb4a1dcb640 1 -- 172.21.15.112:0/867346633 shutdown_connections 2024-09-10T03:41:32.156 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.154+0000 7fb4a1dcb640 1 --2- 172.21.15.112:0/867346633 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb49c101a40 0x7fb49c101ec0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.154+0000 7fb4a1dcb640 1 --2- 172.21.15.112:0/867346633 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb49c100840 0x7fb49c100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.154+0000 7fb4a1dcb640 1 -- 172.21.15.112:0/867346633 >> 172.21.15.112:0/867346633 conn(0x7fb49c0fbff0 msgr2=0x7fb49c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:41:32.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.154+0000 7fb4a1dcb640 1 -- 172.21.15.112:0/867346633 shutdown_connections 2024-09-10T03:41:32.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.154+0000 7fb4a1dcb640 1 -- 172.21.15.112:0/867346633 wait complete. 2024-09-10T03:41:32.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.155+0000 7fb4a1dcb640 1 Processor -- start 2024-09-10T03:41:32.157 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.155+0000 7fb4a1dcb640 1 -- start start 2024-09-10T03:41:32.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.155+0000 7fb4a1dcb640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb49c100840 0x7fb49c1954f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:41:32.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.155+0000 7fb4a1dcb640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb49c195a30 0x7fb49c19aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:41:32.159 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.155+0000 7fb4a1dcb640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb49c108570 con 0x7fb49c100840 2024-09-10T03:41:32.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.155+0000 7fb4a1dcb640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fb49c1083f0 con 0x7fb49c195a30 2024-09-10T03:41:32.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.156+0000 7fb49b7fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb49c100840 0x7fb49c1954f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:41:32.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.156+0000 7fb493fff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb49c195a30 0x7fb49c19aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:41:32.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.156+0000 7fb49b7fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb49c100840 0x7fb49c1954f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:33308/0 (socket says 172.21.15.112:33308) 2024-09-10T03:41:32.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.156+0000 7fb49b7fe640 1 -- 172.21.15.112:0/2850004829 learned_addr learned my addr 172.21.15.112:0/2850004829 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:41:32.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.156+0000 7fb493fff640 1 -- 172.21.15.112:0/2850004829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb49c100840 msgr2=0x7fb49c1954f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:32.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.156+0000 7fb493fff640 1 --2- 172.21.15.112:0/2850004829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb49c100840 0x7fb49c1954f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.156+0000 7fb493fff640 1 -- 172.21.15.112:0/2850004829 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb49c195f60 con 0x7fb49c195a30 2024-09-10T03:41:32.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.156+0000 7fb49b7fe640 1 --2- 172.21.15.112:0/2850004829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb49c100840 0x7fb49c1954f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T03:41:32.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.156+0000 7fb493fff640 1 --2- 172.21.15.112:0/2850004829 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb49c195a30 0x7fb49c19aa30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb484008000 tx=0x7fb484004530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:41:32.160 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.157+0000 7fb4997fa640 1 -- 172.21.15.112:0/2850004829 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb48403f070 con 0x7fb49c195a30 2024-09-10T03:41:32.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.157+0000 7fb4997fa640 1 -- 172.21.15.112:0/2850004829 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb4840049d0 con 0x7fb49c195a30 2024-09-10T03:41:32.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.157+0000 7fb4a1dcb640 1 -- 172.21.15.112:0/2850004829 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb49c19af70 con 0x7fb49c195a30 2024-09-10T03:41:32.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.157+0000 7fb4997fa640 1 -- 172.21.15.112:0/2850004829 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb484035030 con 0x7fb49c195a30 2024-09-10T03:41:32.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.157+0000 7fb4a1dcb640 1 -- 172.21.15.112:0/2850004829 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb49c19b310 con 0x7fb49c195a30 2024-09-10T03:41:32.161 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.158+0000 7fb4a1dcb640 1 -- 172.21.15.112:0/2850004829 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb49c101ec0 con 0x7fb49c195a30 2024-09-10T03:41:32.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.159+0000 7fb4997fa640 1 -- 172.21.15.112:0/2850004829 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fb484042070 con 0x7fb49c195a30 2024-09-10T03:41:32.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.159+0000 7fb4997fa640 1 --2- 172.21.15.112:0/2850004829 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb488070090 0x7fb488072550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:41:32.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.159+0000 7fb4997fa640 1 -- 172.21.15.112:0/2850004829 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7fb4840ae970 con 0x7fb49c195a30 2024-09-10T03:41:32.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.160+0000 7fb49b7fe640 1 --2- 172.21.15.112:0/2850004829 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb488070090 0x7fb488072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:41:32.162 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.160+0000 7fb49b7fe640 1 --2- 172.21.15.112:0/2850004829 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb488070090 0x7fb488072550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb48c005f40 tx=0x7fb48c009500 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:41:32.169 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.168+0000 7fb4997fa640 1 -- 172.21.15.112:0/2850004829 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb484080030 con 0x7fb49c195a30 2024-09-10T03:41:32.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:31 smithi169 ceph-mon[35668]: pgmap v211: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-09-10T03:41:32.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:31 smithi169 ceph-mon[35668]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:41:32.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.421+0000 7fb4a1dcb640 1 -- 172.21.15.112:0/2850004829 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "versions"} v 0) -- 0x7fb49c062df0 con 0x7fb49c195a30 2024-09-10T03:41:32.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.421+0000 7fb4997fa640 1 -- 172.21.15.112:0/2850004829 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) ==== 56+0+633 (secure 0 0 0) 0x7fb484084860 con 0x7fb49c195a30 2024-09-10T03:41:32.423 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-10T03:41:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": { 2024-09-10T03:41:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable)": 2 2024-09-10T03:41:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-10T03:41:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": { 2024-09-10T03:41:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable)": 2 2024-09-10T03:41:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-10T03:41:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": { 2024-09-10T03:41:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable)": 6 2024-09-10T03:41:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-10T03:41:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: "mds": { 2024-09-10T03:41:32.423 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable)": 4 2024-09-10T03:41:32.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-10T03:41:32.424 INFO:teuthology.orchestra.run.smithi112.stdout: "overall": { 2024-09-10T03:41:32.424 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable)": 14 2024-09-10T03:41:32.424 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-10T03:41:32.424 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-10T03:41:32.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.425+0000 7fb4a1dcb640 1 -- 172.21.15.112:0/2850004829 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb488070090 msgr2=0x7fb488072550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:32.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.425+0000 7fb4a1dcb640 1 --2- 172.21.15.112:0/2850004829 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb488070090 0x7fb488072550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb48c005f40 tx=0x7fb48c009500 comp rx=0 tx=0).stop 2024-09-10T03:41:32.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.425+0000 7fb4a1dcb640 1 -- 172.21.15.112:0/2850004829 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb49c195a30 msgr2=0x7fb49c19aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:32.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.425+0000 7fb4a1dcb640 1 --2- 172.21.15.112:0/2850004829 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb49c195a30 0x7fb49c19aa30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb484008000 tx=0x7fb484004530 comp rx=0 tx=0).stop 2024-09-10T03:41:32.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.425+0000 7fb4a1dcb640 1 -- 172.21.15.112:0/2850004829 shutdown_connections 2024-09-10T03:41:32.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.425+0000 7fb4a1dcb640 1 --2- 172.21.15.112:0/2850004829 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fb488070090 0x7fb488072550 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.425+0000 7fb4a1dcb640 1 --2- 172.21.15.112:0/2850004829 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb49c195a30 0x7fb49c19aa30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.425+0000 7fb4a1dcb640 1 --2- 172.21.15.112:0/2850004829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb49c100840 0x7fb49c1954f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.425+0000 7fb4a1dcb640 1 -- 172.21.15.112:0/2850004829 >> 172.21.15.112:0/2850004829 conn(0x7fb49c0fbff0 msgr2=0x7fb49c104fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:41:32.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.426+0000 7fb4a1dcb640 1 -- 172.21.15.112:0/2850004829 shutdown_connections 2024-09-10T03:41:32.427 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.426+0000 7fb4a1dcb640 1 -- 172.21.15.112:0/2850004829 wait complete. 2024-09-10T03:41:32.513 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.512+0000 7f8d1bb91640 1 -- 172.21.15.112:0/2680299027 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d14100840 msgr2=0x7f8d14100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:32.514 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.513+0000 7f8d1bb91640 1 --2- 172.21.15.112:0/2680299027 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d14100840 0x7f8d14100c40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009880 tx=0x7f8d1002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.513+0000 7f8d1bb91640 1 -- 172.21.15.112:0/2680299027 shutdown_connections 2024-09-10T03:41:32.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.513+0000 7f8d1bb91640 1 --2- 172.21.15.112:0/2680299027 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d14101a40 0x7f8d14101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.513+0000 7f8d1bb91640 1 --2- 172.21.15.112:0/2680299027 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d14100840 0x7f8d14100c40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.513+0000 7f8d1bb91640 1 -- 172.21.15.112:0/2680299027 >> 172.21.15.112:0/2680299027 conn(0x7f8d140fbff0 msgr2=0x7f8d140fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:41:32.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.513+0000 7f8d1bb91640 1 -- 172.21.15.112:0/2680299027 shutdown_connections 2024-09-10T03:41:32.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.513+0000 7f8d1bb91640 1 -- 172.21.15.112:0/2680299027 wait complete. 2024-09-10T03:41:32.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.514+0000 7f8d1bb91640 1 Processor -- start 2024-09-10T03:41:32.515 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.514+0000 7f8d1bb91640 1 -- start start 2024-09-10T03:41:32.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.514+0000 7f8d1bb91640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d14100840 0x7f8d141974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:41:32.516 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.514+0000 7f8d1bb91640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d14101a40 0x7f8d14197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:41:32.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.514+0000 7f8d1bb91640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8d14109680 con 0x7f8d14101a40 2024-09-10T03:41:32.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.514+0000 7f8d1bb91640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f8d14109500 con 0x7f8d14100840 2024-09-10T03:41:32.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.515+0000 7f8d1912c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d14101a40 0x7f8d14197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:41:32.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.515+0000 7f8d1912c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d14101a40 0x7f8d14197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:33330/0 (socket says 172.21.15.112:33330) 2024-09-10T03:41:32.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.515+0000 7f8d1912c640 1 -- 172.21.15.112:0/3309893433 learned_addr learned my addr 172.21.15.112:0/3309893433 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:41:32.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.515+0000 7f8d1992d640 1 --2- 172.21.15.112:0/3309893433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d14100840 0x7f8d141974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:41:32.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.515+0000 7f8d1912c640 1 -- 172.21.15.112:0/3309893433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d14100840 msgr2=0x7f8d141974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:32.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.515+0000 7f8d1912c640 1 --2- 172.21.15.112:0/3309893433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d14100840 0x7f8d141974c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.515+0000 7f8d1912c640 1 -- 172.21.15.112:0/3309893433 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d14198080 con 0x7f8d14101a40 2024-09-10T03:41:32.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.515+0000 7f8d1912c640 1 --2- 172.21.15.112:0/3309893433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d14101a40 0x7f8d14197a00 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8d0400e970 tx=0x7f8d0400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:41:32.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.516+0000 7f8d0affd640 1 -- 172.21.15.112:0/3309893433 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8d0400cac0 con 0x7f8d14101a40 2024-09-10T03:41:32.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.516+0000 7f8d1bb91640 1 -- 172.21.15.112:0/3309893433 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d1419cac0 con 0x7f8d14101a40 2024-09-10T03:41:32.517 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.516+0000 7f8d0affd640 1 -- 172.21.15.112:0/3309893433 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8d040054e0 con 0x7f8d14101a40 2024-09-10T03:41:32.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.516+0000 7f8d1bb91640 1 -- 172.21.15.112:0/3309893433 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d1419d050 con 0x7f8d14101a40 2024-09-10T03:41:32.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.517+0000 7f8d0affd640 1 -- 172.21.15.112:0/3309893433 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8d04010430 con 0x7f8d14101a40 2024-09-10T03:41:32.518 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.517+0000 7f8d1bb91640 1 -- 172.21.15.112:0/3309893433 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cf80052c0 con 0x7f8d14101a40 2024-09-10T03:41:32.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.518+0000 7f8d0affd640 1 -- 172.21.15.112:0/3309893433 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f8d040105d0 con 0x7f8d14101a40 2024-09-10T03:41:32.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.519+0000 7f8d0affd640 1 --2- 172.21.15.112:0/3309893433 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f8d0006ded0 0x7f8d00070390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:41:32.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.519+0000 7f8d1992d640 1 --2- 172.21.15.112:0/3309893433 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f8d0006ded0 0x7f8d00070390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:41:32.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.520+0000 7f8d0affd640 1 -- 172.21.15.112:0/3309893433 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7f8d0408e0d0 con 0x7f8d14101a40 2024-09-10T03:41:32.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.520+0000 7f8d1992d640 1 --2- 172.21.15.112:0/3309893433 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f8d0006ded0 0x7f8d00070390 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8d1002a9c0 tx=0x7f8d10000bc0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:41:32.524 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.524+0000 7f8d0affd640 1 -- 172.21.15.112:0/3309893433 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8d0405bd30 con 0x7f8d14101a40 2024-09-10T03:41:32.906 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.905+0000 7f8d1bb91640 1 -- 172.21.15.112:0/3309893433 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) -- 0x7f8cf8000bf0 con 0x7f8d0006ded0 2024-09-10T03:41:32.907 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.906+0000 7f8d0affd640 1 -- 172.21.15.112:0/3309893433 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+439 (secure 0 0 0) 0x7f8cf8000bf0 con 0x7f8d0006ded0 2024-09-10T03:41:32.908 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-10T03:41:32.908 INFO:teuthology.orchestra.run.smithi112.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:2db6858ffc169715474ea237dd8b2beae440a5514686b8202506e12cb7dc1fca", 2024-09-10T03:41:32.908 INFO:teuthology.orchestra.run.smithi112.stdout: "in_progress": true, 2024-09-10T03:41:32.908 INFO:teuthology.orchestra.run.smithi112.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-10T03:41:32.908 INFO:teuthology.orchestra.run.smithi112.stdout: "services_complete": [], 2024-09-10T03:41:32.908 INFO:teuthology.orchestra.run.smithi112.stdout: "progress": "0/2 daemons upgraded", 2024-09-10T03:41:32.908 INFO:teuthology.orchestra.run.smithi112.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:2db6858ffc169715474ea237dd8b2beae440a5514686b8202506e12cb7dc1fca image on host smithi169", 2024-09-10T03:41:32.908 INFO:teuthology.orchestra.run.smithi112.stdout: "is_paused": false 2024-09-10T03:41:32.908 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-10T03:41:32.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.909+0000 7f8d1bb91640 1 -- 172.21.15.112:0/3309893433 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f8d0006ded0 msgr2=0x7f8d00070390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:32.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.909+0000 7f8d1bb91640 1 --2- 172.21.15.112:0/3309893433 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f8d0006ded0 0x7f8d00070390 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8d1002a9c0 tx=0x7f8d10000bc0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.909+0000 7f8d1bb91640 1 -- 172.21.15.112:0/3309893433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d14101a40 msgr2=0x7f8d14197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:41:32.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.909+0000 7f8d1bb91640 1 --2- 172.21.15.112:0/3309893433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d14101a40 0x7f8d14197a00 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8d0400e970 tx=0x7f8d0400ed70 comp rx=0 tx=0).stop 2024-09-10T03:41:32.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.910+0000 7f8d1bb91640 1 -- 172.21.15.112:0/3309893433 shutdown_connections 2024-09-10T03:41:32.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.910+0000 7f8d1bb91640 1 --2- 172.21.15.112:0/3309893433 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f8d0006ded0 0x7f8d00070390 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.911 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.910+0000 7f8d1bb91640 1 --2- 172.21.15.112:0/3309893433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d14101a40 0x7f8d14197a00 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.910+0000 7f8d1bb91640 1 --2- 172.21.15.112:0/3309893433 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8d14100840 0x7f8d141974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:41:32.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.910+0000 7f8d1bb91640 1 -- 172.21.15.112:0/3309893433 >> 172.21.15.112:0/3309893433 conn(0x7f8d140fbff0 msgr2=0x7f8d14069b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:41:32.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.910+0000 7f8d1bb91640 1 -- 172.21.15.112:0/3309893433 shutdown_connections 2024-09-10T03:41:32.912 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:41:32.910+0000 7f8d1bb91640 1 -- 172.21.15.112:0/3309893433 wait complete. 2024-09-10T03:41:33.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:32 smithi169 ceph-mon[35668]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:41:33.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:32 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:41:33.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:32 smithi169 ceph-mon[35668]: Upgrade: Target is version 19.3.0-4885-g3a5e3f16 (squid) 2024-09-10T03:41:33.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:32 smithi169 ceph-mon[35668]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:2db6858ffc169715474ea237dd8b2beae440a5514686b8202506e12cb7dc1fca, digests ['quay.ceph.io/ceph-ci/ceph@sha256:2db6858ffc169715474ea237dd8b2beae440a5514686b8202506e12cb7dc1fca'] 2024-09-10T03:41:33.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:32 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:41:33.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:32 smithi169 ceph-mon[35668]: Upgrade: Need to upgrade myself (mgr.smithi112.pqrlrb) 2024-09-10T03:41:33.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:32 smithi169 ceph-mon[35668]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:41:33.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:32 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/2850004829' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-10T03:41:33.328 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:32 smithi112 ceph-mon[27737]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:41:33.328 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:32 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:41:33.328 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:32 smithi112 ceph-mon[27737]: Upgrade: Target is version 19.3.0-4885-g3a5e3f16 (squid) 2024-09-10T03:41:33.328 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:32 smithi112 ceph-mon[27737]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:2db6858ffc169715474ea237dd8b2beae440a5514686b8202506e12cb7dc1fca, digests ['quay.ceph.io/ceph-ci/ceph@sha256:2db6858ffc169715474ea237dd8b2beae440a5514686b8202506e12cb7dc1fca'] 2024-09-10T03:41:33.328 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:32 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:41:33.328 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:32 smithi112 ceph-mon[27737]: Upgrade: Need to upgrade myself (mgr.smithi112.pqrlrb) 2024-09-10T03:41:33.328 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:32 smithi112 ceph-mon[27737]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:41:33.329 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:32 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/2850004829' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-10T03:41:34.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:33 smithi169 ceph-mon[35668]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:2db6858ffc169715474ea237dd8b2beae440a5514686b8202506e12cb7dc1fca on smithi169 2024-09-10T03:41:34.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:33 smithi169 ceph-mon[35668]: pgmap v212: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-09-10T03:41:34.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:33 smithi169 ceph-mon[35668]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:41:34.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:33 smithi112 ceph-mon[27737]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:2db6858ffc169715474ea237dd8b2beae440a5514686b8202506e12cb7dc1fca on smithi169 2024-09-10T03:41:34.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:33 smithi112 ceph-mon[27737]: pgmap v212: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-09-10T03:41:34.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:33 smithi112 ceph-mon[27737]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:41:35.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:34 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:41:35.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:34 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:41:36.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:35 smithi169 ceph-mon[35668]: pgmap v213: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-09-10T03:41:36.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:35 smithi112 ceph-mon[27737]: pgmap v213: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-09-10T03:41:38.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:37 smithi169 ceph-mon[35668]: pgmap v214: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-09-10T03:41:38.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:37 smithi112 ceph-mon[27737]: pgmap v214: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-09-10T03:41:40.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:40 smithi169 ceph-mon[35668]: pgmap v215: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-09-10T03:41:40.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:40 smithi112 ceph-mon[27737]: pgmap v215: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-09-10T03:41:42.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:42 smithi169 ceph-mon[35668]: pgmap v216: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-09-10T03:41:42.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:42 smithi112 ceph-mon[27737]: pgmap v216: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-09-10T03:41:44.006 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:44 smithi169 ceph-mon[35668]: pgmap v217: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:44.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:44 smithi112 ceph-mon[27737]: pgmap v217: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:46.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:46 smithi169 ceph-mon[35668]: pgmap v218: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:46.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:46 smithi112 ceph-mon[27737]: pgmap v218: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:48.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:48 smithi169 ceph-mon[35668]: pgmap v219: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:48.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:48 smithi112 ceph-mon[27737]: pgmap v219: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:50.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:50 smithi169 ceph-mon[35668]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:50.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:50 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:41:50.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:50 smithi112 ceph-mon[27737]: pgmap v220: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:50.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:50 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:41:51.425 INFO:tasks.workunit.client.0.smithi112.stderr:Note: switching to '3a5e3f16720f432aea965c40a8f6eb480095ec6c'. 2024-09-10T03:41:51.425 INFO:tasks.workunit.client.0.smithi112.stderr: 2024-09-10T03:41:51.425 INFO:tasks.workunit.client.0.smithi112.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-10T03:41:51.425 INFO:tasks.workunit.client.0.smithi112.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-10T03:41:51.425 INFO:tasks.workunit.client.0.smithi112.stderr:state without impacting any branches by switching back to a branch. 2024-09-10T03:41:51.425 INFO:tasks.workunit.client.0.smithi112.stderr: 2024-09-10T03:41:51.425 INFO:tasks.workunit.client.0.smithi112.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-10T03:41:51.425 INFO:tasks.workunit.client.0.smithi112.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-10T03:41:51.425 INFO:tasks.workunit.client.0.smithi112.stderr: 2024-09-10T03:41:51.425 INFO:tasks.workunit.client.0.smithi112.stderr: git switch -c 2024-09-10T03:41:51.426 INFO:tasks.workunit.client.0.smithi112.stderr: 2024-09-10T03:41:51.426 INFO:tasks.workunit.client.0.smithi112.stderr:Or undo this operation with: 2024-09-10T03:41:51.426 INFO:tasks.workunit.client.0.smithi112.stderr: 2024-09-10T03:41:51.426 INFO:tasks.workunit.client.0.smithi112.stderr: git switch - 2024-09-10T03:41:51.426 INFO:tasks.workunit.client.0.smithi112.stderr: 2024-09-10T03:41:51.426 INFO:tasks.workunit.client.0.smithi112.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-10T03:41:51.426 INFO:tasks.workunit.client.0.smithi112.stderr: 2024-09-10T03:41:51.426 INFO:tasks.workunit.client.0.smithi112.stderr:HEAD is now at 3a5e3f16720 Merge branch 'cephadm-enable-module-bootstrap-retry' of https://github.com/adk3798/ceph into wip-adk-testing-2024-09-09-1904 2024-09-10T03:41:51.430 DEBUG:teuthology.orchestra.run.smithi112:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-10T03:41:51.559 INFO:tasks.workunit.client.0.smithi112.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-10T03:41:51.561 INFO:tasks.workunit.client.0.smithi112.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-10T03:41:51.562 INFO:tasks.workunit.client.0.smithi112.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-10T03:41:52.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:52 smithi169 ceph-mon[35668]: pgmap v221: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:52.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:52 smithi112 ceph-mon[27737]: pgmap v221: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:52.706 INFO:tasks.workunit.client.1.smithi169.stderr:Updating files: 87% (11064/12697) Updating files: 88% (11174/12697) Updating files: 89% (11301/12697) Updating files: 90% (11428/12697) Updating files: 91% (11555/12697) Updating files: 92% (11682/12697) Updating files: 93% (11809/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-09-10T03:41:53.576 INFO:tasks.workunit.client.1.smithi169.stderr:Note: switching to '3a5e3f16720f432aea965c40a8f6eb480095ec6c'. 2024-09-10T03:41:53.576 INFO:tasks.workunit.client.1.smithi169.stderr: 2024-09-10T03:41:53.576 INFO:tasks.workunit.client.1.smithi169.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-10T03:41:53.577 INFO:tasks.workunit.client.1.smithi169.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-10T03:41:53.577 INFO:tasks.workunit.client.1.smithi169.stderr:state without impacting any branches by switching back to a branch. 2024-09-10T03:41:53.577 INFO:tasks.workunit.client.1.smithi169.stderr: 2024-09-10T03:41:53.577 INFO:tasks.workunit.client.1.smithi169.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-10T03:41:53.577 INFO:tasks.workunit.client.1.smithi169.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-10T03:41:53.577 INFO:tasks.workunit.client.1.smithi169.stderr: 2024-09-10T03:41:53.577 INFO:tasks.workunit.client.1.smithi169.stderr: git switch -c 2024-09-10T03:41:53.577 INFO:tasks.workunit.client.1.smithi169.stderr: 2024-09-10T03:41:53.577 INFO:tasks.workunit.client.1.smithi169.stderr:Or undo this operation with: 2024-09-10T03:41:53.577 INFO:tasks.workunit.client.1.smithi169.stderr: 2024-09-10T03:41:53.578 INFO:tasks.workunit.client.1.smithi169.stderr: git switch - 2024-09-10T03:41:53.578 INFO:tasks.workunit.client.1.smithi169.stderr: 2024-09-10T03:41:53.578 INFO:tasks.workunit.client.1.smithi169.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-10T03:41:53.578 INFO:tasks.workunit.client.1.smithi169.stderr: 2024-09-10T03:41:53.578 INFO:tasks.workunit.client.1.smithi169.stderr:HEAD is now at 3a5e3f16720 Merge branch 'cephadm-enable-module-bootstrap-retry' of https://github.com/adk3798/ceph into wip-adk-testing-2024-09-09-1904 2024-09-10T03:41:53.582 DEBUG:teuthology.orchestra.run.smithi169:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-09-10T03:41:53.646 INFO:tasks.workunit.client.1.smithi169.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-10T03:41:53.648 INFO:tasks.workunit.client.1.smithi169.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-09-10T03:41:53.648 INFO:tasks.workunit.client.1.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-10T03:41:54.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:54 smithi169 ceph-mon[35668]: pgmap v222: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:54.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:54 smithi112 ceph-mon[27737]: pgmap v222: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:54.335 INFO:tasks.workunit.client.1.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-10T03:41:54.529 INFO:tasks.workunit.client.1.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-10T03:41:54.568 INFO:tasks.workunit.client.1.smithi169.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-09-10T03:41:54.570 INFO:tasks.workunit.client.1.smithi169.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-09-10T03:41:54.570 INFO:tasks.workunit.client.1.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-10T03:41:54.609 INFO:tasks.workunit.client.1.smithi169.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-09-10T03:41:54.612 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-10T03:41:54.612 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-09-10T03:41:54.670 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-09-10T03:41:54.672 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-09-10T03:41:54.672 DEBUG:teuthology.orchestra.run.smithi169:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3a5e3f16720f432aea965c40a8f6eb480095ec6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-09-10T03:41:54.734 INFO:tasks.workunit.client.1.smithi169.stderr:+ mkdir -p fsstress 2024-09-10T03:41:54.736 INFO:tasks.workunit.client.1.smithi169.stderr:+ pushd fsstress 2024-09-10T03:41:54.736 INFO:tasks.workunit.client.1.smithi169.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-09-10T03:41:54.736 INFO:tasks.workunit.client.1.smithi169.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-09-10T03:41:55.255 INFO:tasks.workunit.client.1.smithi169.stderr:+ tar xzf ltp-full.tgz 2024-09-10T03:41:56.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:56 smithi169 ceph-mon[35668]: pgmap v223: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:56.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:56 smithi112 ceph-mon[27737]: pgmap v223: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:41:58.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:41:58 smithi169 ceph-mon[35668]: pgmap v224: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-09-10T03:41:58.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:41:58 smithi112 ceph-mon[27737]: pgmap v224: 65 pgs: 65 active+clean; 588 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-09-10T03:41:59.270 INFO:tasks.workunit.client.0.smithi112.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-10T03:41:59.743 INFO:tasks.workunit.client.0.smithi112.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-10T03:41:59.793 INFO:tasks.workunit.client.0.smithi112.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-10T03:41:59.794 INFO:tasks.workunit.client.0.smithi112.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-10T03:41:59.794 INFO:tasks.workunit.client.0.smithi112.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-10T03:41:59.879 INFO:tasks.workunit.client.0.smithi112.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-10T03:41:59.882 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T03:41:59.883 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-10T03:41:59.899 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-09-10T03:41:59.901 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-09-10T03:41:59.902 DEBUG:teuthology.orchestra.run.smithi112:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3a5e3f16720f432aea965c40a8f6eb480095ec6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-09-10T03:41:59.964 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir -p fsstress 2024-09-10T03:41:59.967 INFO:tasks.workunit.client.0.smithi112.stderr:+ pushd fsstress 2024-09-10T03:41:59.967 INFO:tasks.workunit.client.0.smithi112.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-09-10T03:41:59.968 INFO:tasks.workunit.client.0.smithi112.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-09-10T03:42:00.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:00 smithi169 ceph-mon[35668]: pgmap v225: 65 pgs: 65 active+clean; 588 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-09-10T03:42:00.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:00 smithi112 ceph-mon[27737]: pgmap v225: 65 pgs: 65 active+clean; 588 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2024-09-10T03:42:02.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:02 smithi169 ceph-mon[35668]: pgmap v226: 65 pgs: 65 active+clean; 23 MiB data, 210 MiB used, 536 GiB / 536 GiB avail; 1.9 MiB/s wr, 86 op/s 2024-09-10T03:42:02.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:02 smithi112 ceph-mon[27737]: pgmap v226: 65 pgs: 65 active+clean; 23 MiB data, 210 MiB used, 536 GiB / 536 GiB avail; 1.9 MiB/s wr, 86 op/s 2024-09-10T03:42:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:02.998+0000 7fdfb33d7640 1 -- 172.21.15.112:0/2695963678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfac101a40 msgr2=0x7fdfac101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:02.998+0000 7fdfb33d7640 1 --2- 172.21.15.112:0/2695963678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfac101a40 0x7fdfac101ec0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c009880 tx=0x7fdf9c02a4a0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:02.999+0000 7fdfb33d7640 1 -- 172.21.15.112:0/2695963678 shutdown_connections 2024-09-10T03:42:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:02.999+0000 7fdfb33d7640 1 --2- 172.21.15.112:0/2695963678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfac101a40 0x7fdfac101ec0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:02.999+0000 7fdfb33d7640 1 --2- 172.21.15.112:0/2695963678 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdfac100840 0x7fdfac100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:02.999+0000 7fdfb33d7640 1 -- 172.21.15.112:0/2695963678 >> 172.21.15.112:0/2695963678 conn(0x7fdfac0fbff0 msgr2=0x7fdfac0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:42:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:02.999+0000 7fdfb33d7640 1 -- 172.21.15.112:0/2695963678 shutdown_connections 2024-09-10T03:42:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:02.999+0000 7fdfb33d7640 1 -- 172.21.15.112:0/2695963678 wait complete. 2024-09-10T03:42:03.003 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:02.999+0000 7fdfb33d7640 1 Processor -- start 2024-09-10T03:42:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.000+0000 7fdfb33d7640 1 -- start start 2024-09-10T03:42:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.000+0000 7fdfb33d7640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdfac100840 0x7fdfac1974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.000+0000 7fdfb33d7640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfac101a40 0x7fdfac197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.000+0000 7fdfb33d7640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdfac109680 con 0x7fdfac101a40 2024-09-10T03:42:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.000+0000 7fdfb33d7640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fdfac109500 con 0x7fdfac100840 2024-09-10T03:42:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.000+0000 7fdfb0972640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfac101a40 0x7fdfac197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:42:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.000+0000 7fdfb0972640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfac101a40 0x7fdfac197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:48858/0 (socket says 172.21.15.112:48858) 2024-09-10T03:42:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.000+0000 7fdfb0972640 1 -- 172.21.15.112:0/2736926781 learned_addr learned my addr 172.21.15.112:0/2736926781 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:42:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.000+0000 7fdfb1173640 1 --2- 172.21.15.112:0/2736926781 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdfac100840 0x7fdfac1974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:42:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.000+0000 7fdfb0972640 1 -- 172.21.15.112:0/2736926781 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdfac100840 msgr2=0x7fdfac1974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.000+0000 7fdfb0972640 1 --2- 172.21.15.112:0/2736926781 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdfac100840 0x7fdfac1974d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.000+0000 7fdfb0972640 1 -- 172.21.15.112:0/2736926781 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfac198090 con 0x7fdfac101a40 2024-09-10T03:42:03.004 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.000+0000 7fdfb1173640 1 --2- 172.21.15.112:0/2736926781 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdfac100840 0x7fdfac1974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T03:42:03.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.001+0000 7fdfb0972640 1 --2- 172.21.15.112:0/2736926781 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfac101a40 0x7fdfac197a10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c006010 tx=0x7fdf9c004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:42:03.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.001+0000 7fdfa27fc640 1 -- 172.21.15.112:0/2736926781 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fdf9c03f070 con 0x7fdfac101a40 2024-09-10T03:42:03.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.001+0000 7fdfb33d7640 1 -- 172.21.15.112:0/2736926781 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfac19ca70 con 0x7fdfac101a40 2024-09-10T03:42:03.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.001+0000 7fdfb33d7640 1 -- 172.21.15.112:0/2736926781 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfac19d000 con 0x7fdfac101a40 2024-09-10T03:42:03.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.001+0000 7fdfa27fc640 1 -- 172.21.15.112:0/2736926781 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdf9c0049d0 con 0x7fdfac101a40 2024-09-10T03:42:03.005 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.001+0000 7fdfa27fc640 1 -- 172.21.15.112:0/2736926781 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fdf9c035030 con 0x7fdfac101a40 2024-09-10T03:42:03.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.002+0000 7fdfb33d7640 1 -- 172.21.15.112:0/2736926781 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfac100c40 con 0x7fdfac101a40 2024-09-10T03:42:03.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.002+0000 7fdfa27fc640 1 -- 172.21.15.112:0/2736926781 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fdf9c042070 con 0x7fdfac101a40 2024-09-10T03:42:03.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.003+0000 7fdfa27fc640 1 --2- 172.21.15.112:0/2736926781 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fdf9806de30 0x7fdf980702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:03.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.003+0000 7fdfa27fc640 1 -- 172.21.15.112:0/2736926781 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7fdf9c0ae750 con 0x7fdfac101a40 2024-09-10T03:42:03.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.003+0000 7fdfb1173640 1 --2- 172.21.15.112:0/2736926781 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fdf9806de30 0x7fdf980702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:42:03.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.003+0000 7fdfb1173640 1 --2- 172.21.15.112:0/2736926781 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fdf9806de30 0x7fdf980702f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdfac1018a0 tx=0x7fdfa8005f90 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:42:03.009 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.008+0000 7fdfa27fc640 1 -- 172.21.15.112:0/2736926781 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdf9c07ff40 con 0x7fdfac101a40 2024-09-10T03:42:03.213 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.211+0000 7fdfb33d7640 1 -- 172.21.15.112:0/2736926781 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) -- 0x7fdfac06a2f0 con 0x7fdf9806de30 2024-09-10T03:42:03.214 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.212+0000 7fdfa27fc640 1 -- 172.21.15.112:0/2736926781 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+341 (secure 0 0 0) 0x7fdfac06a2f0 con 0x7fdf9806de30 2024-09-10T03:42:03.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.215+0000 7fdfb33d7640 1 -- 172.21.15.112:0/2736926781 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fdf9806de30 msgr2=0x7fdf980702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:03.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.215+0000 7fdfb33d7640 1 --2- 172.21.15.112:0/2736926781 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fdf9806de30 0x7fdf980702f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdfac1018a0 tx=0x7fdfa8005f90 comp rx=0 tx=0).stop 2024-09-10T03:42:03.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.215+0000 7fdfb33d7640 1 -- 172.21.15.112:0/2736926781 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfac101a40 msgr2=0x7fdfac197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:03.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.215+0000 7fdfb33d7640 1 --2- 172.21.15.112:0/2736926781 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfac101a40 0x7fdfac197a10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c006010 tx=0x7fdf9c004570 comp rx=0 tx=0).stop 2024-09-10T03:42:03.217 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.215+0000 7fdfb33d7640 1 -- 172.21.15.112:0/2736926781 shutdown_connections 2024-09-10T03:42:03.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.215+0000 7fdfb33d7640 1 --2- 172.21.15.112:0/2736926781 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fdf9806de30 0x7fdf980702f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.215+0000 7fdfb33d7640 1 --2- 172.21.15.112:0/2736926781 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfac101a40 0x7fdfac197a10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.215+0000 7fdfb33d7640 1 --2- 172.21.15.112:0/2736926781 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdfac100840 0x7fdfac1974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.215+0000 7fdfb33d7640 1 -- 172.21.15.112:0/2736926781 >> 172.21.15.112:0/2736926781 conn(0x7fdfac0fbff0 msgr2=0x7fdfac069be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:42:03.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.215+0000 7fdfb33d7640 1 -- 172.21.15.112:0/2736926781 shutdown_connections 2024-09-10T03:42:03.218 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.215+0000 7fdfb33d7640 1 -- 172.21.15.112:0/2736926781 wait complete. 2024-09-10T03:42:03.226 INFO:teuthology.orchestra.run.smithi112.stdout:true 2024-09-10T03:42:03.314 INFO:tasks.workunit.client.0.smithi112.stderr:+ tar xzf ltp-full.tgz 2024-09-10T03:42:03.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.313+0000 7f1299941640 1 -- 172.21.15.112:0/3068696015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12940761a0 msgr2=0x7f1294074600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:03.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.313+0000 7f1299941640 1 --2- 172.21.15.112:0/3068696015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12940761a0 0x7f1294074600 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f1284009880 tx=0x7f128402a4a0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.313+0000 7f1299941640 1 -- 172.21.15.112:0/3068696015 shutdown_connections 2024-09-10T03:42:03.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.313+0000 7f1299941640 1 --2- 172.21.15.112:0/3068696015 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1294074b40 0x7f1294074fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.313+0000 7f1299941640 1 --2- 172.21.15.112:0/3068696015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12940761a0 0x7f1294074600 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.313+0000 7f1299941640 1 -- 172.21.15.112:0/3068696015 >> 172.21.15.112:0/3068696015 conn(0x7f12940fd0f0 msgr2=0x7f12940ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:42:03.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.313+0000 7f1299941640 1 -- 172.21.15.112:0/3068696015 shutdown_connections 2024-09-10T03:42:03.315 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.313+0000 7f1299941640 1 -- 172.21.15.112:0/3068696015 wait complete. 2024-09-10T03:42:03.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.314+0000 7f1299941640 1 Processor -- start 2024-09-10T03:42:03.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.314+0000 7f1299941640 1 -- start start 2024-09-10T03:42:03.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.314+0000 7f1299941640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1294074b40 0x7f1294199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:03.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.314+0000 7f1299941640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12940761a0 0x7f1294199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:03.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.314+0000 7f1299941640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f129410c990 con 0x7f1294074b40 2024-09-10T03:42:03.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.314+0000 7f1299941640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f129410c810 con 0x7f12940761a0 2024-09-10T03:42:03.317 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.314+0000 7f1292ffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1294074b40 0x7f1294199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:42:03.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.314+0000 7f1292ffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1294074b40 0x7f1294199670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:48890/0 (socket says 172.21.15.112:48890) 2024-09-10T03:42:03.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.314+0000 7f1292ffd640 1 -- 172.21.15.112:0/89357289 learned_addr learned my addr 172.21.15.112:0/89357289 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:42:03.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.314+0000 7f12927fc640 1 --2- 172.21.15.112:0/89357289 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12940761a0 0x7f1294199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:42:03.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.315+0000 7f1292ffd640 1 -- 172.21.15.112:0/89357289 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12940761a0 msgr2=0x7f1294199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:03.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.315+0000 7f1292ffd640 1 --2- 172.21.15.112:0/89357289 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12940761a0 0x7f1294199bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.315+0000 7f1292ffd640 1 -- 172.21.15.112:0/89357289 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f129419a230 con 0x7f1294074b40 2024-09-10T03:42:03.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.315+0000 7f1292ffd640 1 --2- 172.21.15.112:0/89357289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1294074b40 0x7f1294199670 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1284005ab0 tx=0x7f12840045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:42:03.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.315+0000 7f129893f640 1 -- 172.21.15.112:0/89357289 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f128403f070 con 0x7f1294074b40 2024-09-10T03:42:03.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.315+0000 7f129893f640 1 -- 172.21.15.112:0/89357289 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1284004a50 con 0x7f1294074b40 2024-09-10T03:42:03.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.315+0000 7f1299941640 1 -- 172.21.15.112:0/89357289 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f129419ec10 con 0x7f1294074b40 2024-09-10T03:42:03.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.315+0000 7f129893f640 1 -- 172.21.15.112:0/89357289 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f1284035030 con 0x7f1294074b40 2024-09-10T03:42:03.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.315+0000 7f1299941640 1 -- 172.21.15.112:0/89357289 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f129419f1a0 con 0x7f1294074b40 2024-09-10T03:42:03.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.316+0000 7f1299941640 1 -- 172.21.15.112:0/89357289 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1294074fa0 con 0x7f1294074b40 2024-09-10T03:42:03.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.317+0000 7f129893f640 1 -- 172.21.15.112:0/89357289 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f1284042070 con 0x7f1294074b40 2024-09-10T03:42:03.319 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.318+0000 7f129893f640 1 --2- 172.21.15.112:0/89357289 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f127c06de30 0x7f127c0702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:03.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.318+0000 7f129893f640 1 -- 172.21.15.112:0/89357289 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7f12840aece0 con 0x7f1294074b40 2024-09-10T03:42:03.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.318+0000 7f12927fc640 1 --2- 172.21.15.112:0/89357289 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f127c06de30 0x7f127c0702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:42:03.320 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.318+0000 7f12927fc640 1 --2- 172.21.15.112:0/89357289 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f127c06de30 0x7f127c0702f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f129419ab20 tx=0x7f1288005f90 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:42:03.326 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.324+0000 7f129893f640 1 -- 172.21.15.112:0/89357289 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f128407c8a0 con 0x7f1294074b40 2024-09-10T03:42:03.578 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.576+0000 7f1299941640 1 -- 172.21.15.112:0/89357289 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) -- 0x7f1294062630 con 0x7f127c06de30 2024-09-10T03:42:03.579 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.577+0000 7f129893f640 1 -- 172.21.15.112:0/89357289 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+341 (secure 0 0 0) 0x7f1294062630 con 0x7f127c06de30 2024-09-10T03:42:03.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.580+0000 7f1299941640 1 -- 172.21.15.112:0/89357289 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f127c06de30 msgr2=0x7f127c0702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:03.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.580+0000 7f1299941640 1 --2- 172.21.15.112:0/89357289 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f127c06de30 0x7f127c0702f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f129419ab20 tx=0x7f1288005f90 comp rx=0 tx=0).stop 2024-09-10T03:42:03.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.580+0000 7f1299941640 1 -- 172.21.15.112:0/89357289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1294074b40 msgr2=0x7f1294199670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:03.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.580+0000 7f1299941640 1 --2- 172.21.15.112:0/89357289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1294074b40 0x7f1294199670 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1284005ab0 tx=0x7f12840045e0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.580+0000 7f1299941640 1 -- 172.21.15.112:0/89357289 shutdown_connections 2024-09-10T03:42:03.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.580+0000 7f1299941640 1 --2- 172.21.15.112:0/89357289 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f127c06de30 0x7f127c0702f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.580+0000 7f1299941640 1 --2- 172.21.15.112:0/89357289 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12940761a0 0x7f1294199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.582 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.580+0000 7f1299941640 1 --2- 172.21.15.112:0/89357289 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1294074b40 0x7f1294199670 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.580+0000 7f1299941640 1 -- 172.21.15.112:0/89357289 >> 172.21.15.112:0/89357289 conn(0x7f12940fd0f0 msgr2=0x7f1294107a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:42:03.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.580+0000 7f1299941640 1 -- 172.21.15.112:0/89357289 shutdown_connections 2024-09-10T03:42:03.583 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.580+0000 7f1299941640 1 -- 172.21.15.112:0/89357289 wait complete. 2024-09-10T03:42:03.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.672+0000 7fe246316640 1 -- 172.21.15.112:0/655903386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe240100220 msgr2=0x7fe240100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:03.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.672+0000 7fe246316640 1 --2- 172.21.15.112:0/655903386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe240100220 0x7fe240100620 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe230009880 tx=0x7fe23002a4a0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.672+0000 7fe246316640 1 -- 172.21.15.112:0/655903386 shutdown_connections 2024-09-10T03:42:03.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.672+0000 7fe246316640 1 --2- 172.21.15.112:0/655903386 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe240100b60 0x7fe240100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.672+0000 7fe246316640 1 --2- 172.21.15.112:0/655903386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe240100220 0x7fe240100620 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.672+0000 7fe246316640 1 -- 172.21.15.112:0/655903386 >> 172.21.15.112:0/655903386 conn(0x7fe2400fbff0 msgr2=0x7fe2400fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:42:03.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.673+0000 7fe246316640 1 -- 172.21.15.112:0/655903386 shutdown_connections 2024-09-10T03:42:03.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.673+0000 7fe246316640 1 -- 172.21.15.112:0/655903386 wait complete. 2024-09-10T03:42:03.675 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.673+0000 7fe246316640 1 Processor -- start 2024-09-10T03:42:03.676 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.673+0000 7fe246316640 1 -- start start 2024-09-10T03:42:03.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.673+0000 7fe246316640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe240100220 0x7fe240199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:03.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.674+0000 7fe246316640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe240100b60 0x7fe240199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:03.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.674+0000 7fe246316640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe24010c9d0 con 0x7fe240100b60 2024-09-10T03:42:03.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.674+0000 7fe246316640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7fe24010c850 con 0x7fe240100220 2024-09-10T03:42:03.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.674+0000 7fe23f7fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe240100b60 0x7fe240199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:42:03.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.674+0000 7fe23ffff640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe240100220 0x7fe240199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:42:03.677 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.674+0000 7fe23f7fe640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe240100b60 0x7fe240199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:48908/0 (socket says 172.21.15.112:48908) 2024-09-10T03:42:03.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.674+0000 7fe23f7fe640 1 -- 172.21.15.112:0/1565233701 learned_addr learned my addr 172.21.15.112:0/1565233701 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:42:03.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.674+0000 7fe23f7fe640 1 -- 172.21.15.112:0/1565233701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe240100220 msgr2=0x7fe240199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:03.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.674+0000 7fe23f7fe640 1 --2- 172.21.15.112:0/1565233701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe240100220 0x7fe240199710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.674+0000 7fe23f7fe640 1 -- 172.21.15.112:0/1565233701 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe24019a2d0 con 0x7fe240100b60 2024-09-10T03:42:03.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.674+0000 7fe23f7fe640 1 --2- 172.21.15.112:0/1565233701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe240100b60 0x7fe240199c50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe23400b420 tx=0x7fe23400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:42:03.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.674+0000 7fe23d7fa640 1 -- 172.21.15.112:0/1565233701 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe234011630 con 0x7fe240100b60 2024-09-10T03:42:03.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.674+0000 7fe23d7fa640 1 -- 172.21.15.112:0/1565233701 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe234016070 con 0x7fe240100b60 2024-09-10T03:42:03.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.674+0000 7fe23d7fa640 1 -- 172.21.15.112:0/1565233701 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe234010d40 con 0x7fe240100b60 2024-09-10T03:42:03.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.675+0000 7fe246316640 1 -- 172.21.15.112:0/1565233701 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe24019ed10 con 0x7fe240100b60 2024-09-10T03:42:03.678 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.675+0000 7fe246316640 1 -- 172.21.15.112:0/1565233701 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe24019f2c0 con 0x7fe240100b60 2024-09-10T03:42:03.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.676+0000 7fe246316640 1 -- 172.21.15.112:0/1565233701 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe240100620 con 0x7fe240100b60 2024-09-10T03:42:03.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.676+0000 7fe23d7fa640 1 -- 172.21.15.112:0/1565233701 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7fe23401b460 con 0x7fe240100b60 2024-09-10T03:42:03.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.677+0000 7fe23d7fa640 1 --2- 172.21.15.112:0/1565233701 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fe22806de30 0x7fe2280702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:03.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.677+0000 7fe23d7fa640 1 -- 172.21.15.112:0/1565233701 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7fe23408d420 con 0x7fe240100b60 2024-09-10T03:42:03.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.677+0000 7fe23ffff640 1 --2- 172.21.15.112:0/1565233701 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fe22806de30 0x7fe2280702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:42:03.679 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.678+0000 7fe23ffff640 1 --2- 172.21.15.112:0/1565233701 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fe22806de30 0x7fe2280702f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe230000c00 tx=0x7fe2300058d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:42:03.683 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.682+0000 7fe23d7fa640 1 -- 172.21.15.112:0/1565233701 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe23405af60 con 0x7fe240100b60 2024-09-10T03:42:03.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:03 smithi112 ceph-mon[27737]: Upgrade: Updating mgr.smithi169.bkwgjl 2024-09-10T03:42:03.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:03 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:03.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:03 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi169.bkwgjl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T03:42:03.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:03 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T03:42:03.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:03 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:42:03.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:03 smithi112 ceph-mon[27737]: Deploying daemon mgr.smithi169.bkwgjl on smithi169 2024-09-10T03:42:03.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:03 smithi112 ceph-mon[27737]: pgmap v227: 65 pgs: 65 active+clean; 31 MiB data, 237 MiB used, 536 GiB / 536 GiB avail; 2.6 MiB/s wr, 152 op/s 2024-09-10T03:42:03.933 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.931+0000 7fe246316640 1 -- 172.21.15.112:0/1565233701 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fe240062630 con 0x7fe22806de30 2024-09-10T03:42:03.951 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.949+0000 7fe23d7fa640 1 -- 172.21.15.112:0/1565233701 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3840 (secure 0 0 0) 0x7fe240062630 con 0x7fe22806de30 2024-09-10T03:42:03.951 INFO:teuthology.orchestra.run.smithi112.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-10T03:42:03.951 INFO:teuthology.orchestra.run.smithi112.stdout:alertmanager.smithi112 smithi112 *:9093,9094 running (5m) 77s ago 6m 26.1M - 0.25.0 c8568f914cd2 28468ac02dbb 2024-09-10T03:42:03.951 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-exporter.smithi112 smithi112 running (7m) 77s ago 7m 8208k - 19.1.1-298-gb53b6d3e 54467abc55f8 0ff34f471a47 2024-09-10T03:42:03.951 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-exporter.smithi169 smithi169 running (6m) 81s ago 6m 5263k - 19.1.1-298-gb53b6d3e 54467abc55f8 f430c6826280 2024-09-10T03:42:03.951 INFO:teuthology.orchestra.run.smithi112.stdout:crash.smithi112 smithi112 running (7m) 77s ago 7m 6899k - 19.1.1-298-gb53b6d3e 54467abc55f8 f7d1df4e46c8 2024-09-10T03:42:03.951 INFO:teuthology.orchestra.run.smithi112.stdout:crash.smithi169 smithi169 running (6m) 81s ago 6m 6899k - 19.1.1-298-gb53b6d3e 54467abc55f8 a27310b2722f 2024-09-10T03:42:03.952 INFO:teuthology.orchestra.run.smithi112.stdout:grafana.smithi112 smithi112 *:3000 running (5m) 77s ago 6m 80.6M - 9.4.12 2bacad6d85d8 444b4b1d4b44 2024-09-10T03:42:03.952 INFO:teuthology.orchestra.run.smithi112.stdout:mds.cephfs.smithi112.ifnhnl smithi112 running (101s) 77s ago 100s 11.7M - 19.1.1-298-gb53b6d3e 54467abc55f8 5737c46b4900 2024-09-10T03:42:03.952 INFO:teuthology.orchestra.run.smithi112.stdout:mds.cephfs.smithi112.uuenbo smithi112 running (94s) 77s ago 94s 16.4M - 19.1.1-298-gb53b6d3e 54467abc55f8 d62acd5a079a 2024-09-10T03:42:03.952 INFO:teuthology.orchestra.run.smithi112.stdout:mds.cephfs.smithi169.gcmhul smithi169 running (92s) 81s ago 91s 13.1M - 19.1.1-298-gb53b6d3e 54467abc55f8 36b459a6aef7 2024-09-10T03:42:03.952 INFO:teuthology.orchestra.run.smithi112.stdout:mds.cephfs.smithi169.wfmonc smithi169 running (98s) 81s ago 98s 14.7M - 19.1.1-298-gb53b6d3e 54467abc55f8 13b086adcdc0 2024-09-10T03:42:03.952 INFO:teuthology.orchestra.run.smithi112.stdout:mgr.smithi112.pqrlrb smithi112 *:9283,8765,8443 running (8m) 77s ago 8m 499M - 19.1.1-298-gb53b6d3e 54467abc55f8 5bcbeef123a8 2024-09-10T03:42:03.952 INFO:teuthology.orchestra.run.smithi112.stdout:mgr.smithi169.bkwgjl smithi169 *:8443,9283,8765 running (6m) 81s ago 6m 441M - 19.1.1-298-gb53b6d3e 54467abc55f8 b78844ecd2af 2024-09-10T03:42:03.952 INFO:teuthology.orchestra.run.smithi112.stdout:mon.smithi112 smithi112 running (8m) 77s ago 8m 47.4M 2048M 19.1.1-298-gb53b6d3e 54467abc55f8 91597bdf09f7 2024-09-10T03:42:03.952 INFO:teuthology.orchestra.run.smithi112.stdout:mon.smithi169 smithi169 running (6m) 81s ago 6m 35.2M 2048M 19.1.1-298-gb53b6d3e 54467abc55f8 38476c275eb5 2024-09-10T03:42:03.952 INFO:teuthology.orchestra.run.smithi112.stdout:node-exporter.smithi112 smithi112 *:9100 running (7m) 77s ago 7m 15.8M - 1.5.0 0da6a335fe13 736b09a514a8 2024-09-10T03:42:03.952 INFO:teuthology.orchestra.run.smithi112.stdout:node-exporter.smithi169 smithi169 *:9100 running (6m) 81s ago 6m 14.8M - 1.5.0 0da6a335fe13 b6d2ee0493db 2024-09-10T03:42:03.952 INFO:teuthology.orchestra.run.smithi112.stdout:osd.0 smithi112 running (5m) 77s ago 5m 56.2M 3946M 19.1.1-298-gb53b6d3e 54467abc55f8 274fcfdbe0d8 2024-09-10T03:42:03.952 INFO:teuthology.orchestra.run.smithi112.stdout:osd.1 smithi112 running (4m) 77s ago 4m 56.6M 3946M 19.1.1-298-gb53b6d3e 54467abc55f8 c48f7eddf6a2 2024-09-10T03:42:03.952 INFO:teuthology.orchestra.run.smithi112.stdout:osd.2 smithi112 running (3m) 77s ago 3m 48.5M 3946M 19.1.1-298-gb53b6d3e 54467abc55f8 2885646ffe21 2024-09-10T03:42:03.953 INFO:teuthology.orchestra.run.smithi112.stdout:osd.3 smithi169 running (3m) 81s ago 3m 45.2M 4970M 19.1.1-298-gb53b6d3e 54467abc55f8 b55e9df427b0 2024-09-10T03:42:03.953 INFO:teuthology.orchestra.run.smithi112.stdout:osd.4 smithi169 running (3m) 81s ago 3m 52.0M 4970M 19.1.1-298-gb53b6d3e 54467abc55f8 f3527032352f 2024-09-10T03:42:03.953 INFO:teuthology.orchestra.run.smithi112.stdout:osd.5 smithi169 running (2m) 81s ago 2m 50.6M 4970M 19.1.1-298-gb53b6d3e 54467abc55f8 b4df34a99808 2024-09-10T03:42:03.953 INFO:teuthology.orchestra.run.smithi112.stdout:prometheus.smithi112 smithi112 *:9095 running (5m) 77s ago 6m 59.1M - 2.43.0 a07b618ecd1d 3cb9cb258756 2024-09-10T03:42:03.953 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.951+0000 7fe246316640 1 -- 172.21.15.112:0/1565233701 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fe22806de30 msgr2=0x7fe2280702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:03.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.951+0000 7fe246316640 1 --2- 172.21.15.112:0/1565233701 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fe22806de30 0x7fe2280702f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe230000c00 tx=0x7fe2300058d0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.951+0000 7fe246316640 1 -- 172.21.15.112:0/1565233701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe240100b60 msgr2=0x7fe240199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:03.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.951+0000 7fe246316640 1 --2- 172.21.15.112:0/1565233701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe240100b60 0x7fe240199c50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe23400b420 tx=0x7fe23400b8e0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.951+0000 7fe246316640 1 -- 172.21.15.112:0/1565233701 shutdown_connections 2024-09-10T03:42:03.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.951+0000 7fe246316640 1 --2- 172.21.15.112:0/1565233701 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7fe22806de30 0x7fe2280702f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.951+0000 7fe246316640 1 --2- 172.21.15.112:0/1565233701 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe240100b60 0x7fe240199c50 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.951+0000 7fe246316640 1 --2- 172.21.15.112:0/1565233701 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe240100220 0x7fe240199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:03.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.951+0000 7fe246316640 1 -- 172.21.15.112:0/1565233701 >> 172.21.15.112:0/1565233701 conn(0x7fe2400fbff0 msgr2=0x7fe240108ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:42:03.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.951+0000 7fe246316640 1 -- 172.21.15.112:0/1565233701 shutdown_connections 2024-09-10T03:42:03.954 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:03.951+0000 7fe246316640 1 -- 172.21.15.112:0/1565233701 wait complete. 2024-09-10T03:42:03.998 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:03 smithi169 ceph-mon[35668]: Upgrade: Updating mgr.smithi169.bkwgjl 2024-09-10T03:42:03.998 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:03 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:03.998 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:03 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi169.bkwgjl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T03:42:03.998 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:03 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T03:42:03.998 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:03 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:42:03.998 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:03 smithi169 ceph-mon[35668]: Deploying daemon mgr.smithi169.bkwgjl on smithi169 2024-09-10T03:42:03.998 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:03 smithi169 ceph-mon[35668]: pgmap v227: 65 pgs: 65 active+clean; 31 MiB data, 237 MiB used, 536 GiB / 536 GiB avail; 2.6 MiB/s wr, 152 op/s 2024-09-10T03:42:04.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.045+0000 7f6c59878640 1 -- 172.21.15.112:0/1106953514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c54100840 msgr2=0x7f6c54100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:04.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.045+0000 7f6c59878640 1 --2- 172.21.15.112:0/1106953514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c54100840 0x7f6c54100c40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6c44009880 tx=0x7f6c4402a4a0 comp rx=0 tx=0).stop 2024-09-10T03:42:04.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.045+0000 7f6c59878640 1 -- 172.21.15.112:0/1106953514 shutdown_connections 2024-09-10T03:42:04.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.045+0000 7f6c59878640 1 --2- 172.21.15.112:0/1106953514 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c54101a40 0x7f6c54101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:04.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.045+0000 7f6c59878640 1 --2- 172.21.15.112:0/1106953514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c54100840 0x7f6c54100c40 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:04.047 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.045+0000 7f6c59878640 1 -- 172.21.15.112:0/1106953514 >> 172.21.15.112:0/1106953514 conn(0x7f6c540fbff0 msgr2=0x7f6c540fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:42:04.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.045+0000 7f6c59878640 1 -- 172.21.15.112:0/1106953514 shutdown_connections 2024-09-10T03:42:04.048 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.045+0000 7f6c59878640 1 -- 172.21.15.112:0/1106953514 wait complete. 2024-09-10T03:42:04.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.046+0000 7f6c59878640 1 Processor -- start 2024-09-10T03:42:04.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.046+0000 7f6c59878640 1 -- start start 2024-09-10T03:42:04.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.046+0000 7f6c59878640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c54100840 0x7f6c5406a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:04.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.046+0000 7f6c59878640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c54101a40 0x7f6c5419c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:04.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.046+0000 7f6c59878640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6c54109660 con 0x7f6c54101a40 2024-09-10T03:42:04.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.046+0000 7f6c59878640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f6c541094e0 con 0x7f6c54100840 2024-09-10T03:42:04.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.046+0000 7f6c4bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c54101a40 0x7f6c5419c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:42:04.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.046+0000 7f6c4bfff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c54101a40 0x7f6c5419c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:48926/0 (socket says 172.21.15.112:48926) 2024-09-10T03:42:04.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.046+0000 7f6c4bfff640 1 -- 172.21.15.112:0/3008344484 learned_addr learned my addr 172.21.15.112:0/3008344484 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:42:04.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.046+0000 7f6c52ffd640 1 --2- 172.21.15.112:0/3008344484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c54100840 0x7f6c5406a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:42:04.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.046+0000 7f6c4bfff640 1 -- 172.21.15.112:0/3008344484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c54100840 msgr2=0x7f6c5406a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:04.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.046+0000 7f6c4bfff640 1 --2- 172.21.15.112:0/3008344484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c54100840 0x7f6c5406a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:04.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.046+0000 7f6c4bfff640 1 -- 172.21.15.112:0/3008344484 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c5419c7b0 con 0x7f6c54101a40 2024-09-10T03:42:04.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.047+0000 7f6c4bfff640 1 --2- 172.21.15.112:0/3008344484 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c54101a40 0x7f6c5419c1f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00d8d0 tx=0x7f6c3c00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:42:04.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.047+0000 7f6c58876640 1 -- 172.21.15.112:0/3008344484 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6c3c017070 con 0x7f6c54101a40 2024-09-10T03:42:04.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.047+0000 7f6c58876640 1 -- 172.21.15.112:0/3008344484 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6c3c010460 con 0x7f6c54101a40 2024-09-10T03:42:04.050 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.047+0000 7f6c59878640 1 -- 172.21.15.112:0/3008344484 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c5419caa0 con 0x7f6c54101a40 2024-09-10T03:42:04.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.047+0000 7f6c58876640 1 -- 172.21.15.112:0/3008344484 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6c3c00f5d0 con 0x7f6c54101a40 2024-09-10T03:42:04.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.047+0000 7f6c59878640 1 -- 172.21.15.112:0/3008344484 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c5419d030 con 0x7f6c54101a40 2024-09-10T03:42:04.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.048+0000 7f6c59878640 1 -- 172.21.15.112:0/3008344484 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c54100c40 con 0x7f6c54101a40 2024-09-10T03:42:04.051 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.049+0000 7f6c58876640 1 -- 172.21.15.112:0/3008344484 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f6c3c009a30 con 0x7f6c54101a40 2024-09-10T03:42:04.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.050+0000 7f6c58876640 1 --2- 172.21.15.112:0/3008344484 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6c4006de90 0x7f6c40070350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:04.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.050+0000 7f6c58876640 1 -- 172.21.15.112:0/3008344484 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7f6c3c08d9c0 con 0x7f6c54101a40 2024-09-10T03:42:04.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.050+0000 7f6c52ffd640 1 --2- 172.21.15.112:0/3008344484 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6c4006de90 0x7f6c40070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:42:04.052 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.051+0000 7f6c52ffd640 1 --2- 172.21.15.112:0/3008344484 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6c4006de90 0x7f6c40070350 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6c44009230 tx=0x7f6c440058d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:42:04.060 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.058+0000 7f6c58876640 1 -- 172.21.15.112:0/3008344484 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6c3c05f1a0 con 0x7f6c54101a40 2024-09-10T03:42:04.366 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.364+0000 7f6c59878640 1 -- 172.21.15.112:0/3008344484 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "versions"} v 0) -- 0x7f6c54062df0 con 0x7f6c54101a40 2024-09-10T03:42:04.367 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.365+0000 7f6c58876640 1 -- 172.21.15.112:0/3008344484 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) ==== 56+0+633 (secure 0 0 0) 0x7f6c3c0639d0 con 0x7f6c54101a40 2024-09-10T03:42:04.367 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-10T03:42:04.367 INFO:teuthology.orchestra.run.smithi112.stdout: "mon": { 2024-09-10T03:42:04.367 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable)": 2 2024-09-10T03:42:04.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-10T03:42:04.367 INFO:teuthology.orchestra.run.smithi112.stdout: "mgr": { 2024-09-10T03:42:04.367 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable)": 2 2024-09-10T03:42:04.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-10T03:42:04.367 INFO:teuthology.orchestra.run.smithi112.stdout: "osd": { 2024-09-10T03:42:04.368 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable)": 6 2024-09-10T03:42:04.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-10T03:42:04.368 INFO:teuthology.orchestra.run.smithi112.stdout: "mds": { 2024-09-10T03:42:04.368 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable)": 4 2024-09-10T03:42:04.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-10T03:42:04.368 INFO:teuthology.orchestra.run.smithi112.stdout: "overall": { 2024-09-10T03:42:04.368 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable)": 14 2024-09-10T03:42:04.368 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-10T03:42:04.368 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-10T03:42:04.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.367+0000 7f6c59878640 1 -- 172.21.15.112:0/3008344484 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6c4006de90 msgr2=0x7f6c40070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:04.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.367+0000 7f6c59878640 1 --2- 172.21.15.112:0/3008344484 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6c4006de90 0x7f6c40070350 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6c44009230 tx=0x7f6c440058d0 comp rx=0 tx=0).stop 2024-09-10T03:42:04.369 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.367+0000 7f6c59878640 1 -- 172.21.15.112:0/3008344484 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c54101a40 msgr2=0x7f6c5419c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:04.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.367+0000 7f6c59878640 1 --2- 172.21.15.112:0/3008344484 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c54101a40 0x7f6c5419c1f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00d8d0 tx=0x7f6c3c00dd90 comp rx=0 tx=0).stop 2024-09-10T03:42:04.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.367+0000 7f6c59878640 1 -- 172.21.15.112:0/3008344484 shutdown_connections 2024-09-10T03:42:04.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.367+0000 7f6c59878640 1 --2- 172.21.15.112:0/3008344484 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f6c4006de90 0x7f6c40070350 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:04.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.367+0000 7f6c59878640 1 --2- 172.21.15.112:0/3008344484 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6c54101a40 0x7f6c5419c1f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:04.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.367+0000 7f6c59878640 1 --2- 172.21.15.112:0/3008344484 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6c54100840 0x7f6c5406a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:04.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.367+0000 7f6c59878640 1 -- 172.21.15.112:0/3008344484 >> 172.21.15.112:0/3008344484 conn(0x7f6c540fbff0 msgr2=0x7f6c54104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:42:04.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.367+0000 7f6c59878640 1 -- 172.21.15.112:0/3008344484 shutdown_connections 2024-09-10T03:42:04.370 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.367+0000 7f6c59878640 1 -- 172.21.15.112:0/3008344484 wait complete. 2024-09-10T03:42:04.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.463+0000 7f4c194c2640 1 -- 172.21.15.112:0/1134855591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c14101a40 msgr2=0x7f4c14101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:04.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.463+0000 7f4c194c2640 1 --2- 172.21.15.112:0/1134855591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c14101a40 0x7f4c14101ec0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc009880 tx=0x7f4bfc02a4a0 comp rx=0 tx=0).stop 2024-09-10T03:42:04.465 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.463+0000 7f4c194c2640 1 -- 172.21.15.112:0/1134855591 shutdown_connections 2024-09-10T03:42:04.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.463+0000 7f4c194c2640 1 --2- 172.21.15.112:0/1134855591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c14101a40 0x7f4c14101ec0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:04.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.463+0000 7f4c194c2640 1 --2- 172.21.15.112:0/1134855591 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c14100840 0x7f4c14100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:04.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.463+0000 7f4c194c2640 1 -- 172.21.15.112:0/1134855591 >> 172.21.15.112:0/1134855591 conn(0x7f4c140fbff0 msgr2=0x7f4c140fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:42:04.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.463+0000 7f4c194c2640 1 -- 172.21.15.112:0/1134855591 shutdown_connections 2024-09-10T03:42:04.466 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.463+0000 7f4c194c2640 1 -- 172.21.15.112:0/1134855591 wait complete. 2024-09-10T03:42:04.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.464+0000 7f4c194c2640 1 Processor -- start 2024-09-10T03:42:04.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.464+0000 7f4c194c2640 1 -- start start 2024-09-10T03:42:04.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.464+0000 7f4c194c2640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c14100840 0x7f4c1406a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:04.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.464+0000 7f4c194c2640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c14101a40 0x7f4c1419c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:04.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.464+0000 7f4c194c2640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4c14109660 con 0x7f4c14100840 2024-09-10T03:42:04.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.464+0000 7f4c194c2640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f4c141094e0 con 0x7f4c14101a40 2024-09-10T03:42:04.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.464+0000 7f4c12ffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c14100840 0x7f4c1406a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:42:04.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.465+0000 7f4c12ffd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c14100840 0x7f4c1406a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:48954/0 (socket says 172.21.15.112:48954) 2024-09-10T03:42:04.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.465+0000 7f4c12ffd640 1 -- 172.21.15.112:0/1416309841 learned_addr learned my addr 172.21.15.112:0/1416309841 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:42:04.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.465+0000 7f4c0a7fc640 1 --2- 172.21.15.112:0/1416309841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c14101a40 0x7f4c1419c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:42:04.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.465+0000 7f4c12ffd640 1 -- 172.21.15.112:0/1416309841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c14101a40 msgr2=0x7f4c1419c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:04.468 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.465+0000 7f4c12ffd640 1 --2- 172.21.15.112:0/1416309841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c14101a40 0x7f4c1419c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:04.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.465+0000 7f4c12ffd640 1 -- 172.21.15.112:0/1416309841 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c1419c710 con 0x7f4c14100840 2024-09-10T03:42:04.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.465+0000 7f4c12ffd640 1 --2- 172.21.15.112:0/1416309841 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c14100840 0x7f4c1406a620 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400c920 tx=0x7f4c0400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:42:04.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.465+0000 7f4c10ff9640 1 -- 172.21.15.112:0/1416309841 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f4c04007a10 con 0x7f4c14100840 2024-09-10T03:42:04.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.465+0000 7f4c10ff9640 1 -- 172.21.15.112:0/1416309841 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4c04007bb0 con 0x7f4c14100840 2024-09-10T03:42:04.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.465+0000 7f4c194c2640 1 -- 172.21.15.112:0/1416309841 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c1419ca00 con 0x7f4c14100840 2024-09-10T03:42:04.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.465+0000 7f4c10ff9640 1 -- 172.21.15.112:0/1416309841 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f4c0400f450 con 0x7f4c14100840 2024-09-10T03:42:04.469 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.465+0000 7f4c194c2640 1 -- 172.21.15.112:0/1416309841 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c1419cf60 con 0x7f4c14100840 2024-09-10T03:42:04.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.467+0000 7f4c194c2640 1 -- 172.21.15.112:0/1416309841 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c14109e40 con 0x7f4c14100840 2024-09-10T03:42:04.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.467+0000 7f4c10ff9640 1 -- 172.21.15.112:0/1416309841 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 92205+0+0 (secure 0 0 0) 0x7f4c04003ea0 con 0x7f4c14100840 2024-09-10T03:42:04.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.468+0000 7f4c10ff9640 1 --2- 172.21.15.112:0/1416309841 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f4c0006de80 0x7f4c00070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:04.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.468+0000 7f4c10ff9640 1 -- 172.21.15.112:0/1416309841 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5554+0+0 (secure 0 0 0) 0x7f4c0408db40 con 0x7f4c14100840 2024-09-10T03:42:04.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.468+0000 7f4c0a7fc640 1 --2- 172.21.15.112:0/1416309841 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f4c0006de80 0x7f4c00070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:42:04.470 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.468+0000 7f4c0a7fc640 1 --2- 172.21.15.112:0/1416309841 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f4c0006de80 0x7f4c00070340 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc000c00 tx=0x7f4bfc005a20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:42:04.474 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.472+0000 7f4c10ff9640 1 -- 172.21.15.112:0/1416309841 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4c0405b700 con 0x7f4c14100840 2024-09-10T03:42:04.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.733+0000 7f4c194c2640 1 -- 172.21.15.112:0/1416309841 --> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) -- 0x7f4c14062630 con 0x7f4c0006de80 2024-09-10T03:42:04.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.734+0000 7f4c10ff9640 1 -- 172.21.15.112:0/1416309841 <== mgr.14221 v2:172.21.15.112:6800/3600199018 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+341 (secure 0 0 0) 0x7f4c14062630 con 0x7f4c0006de80 2024-09-10T03:42:04.736 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-10T03:42:04.736 INFO:teuthology.orchestra.run.smithi112.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:2db6858ffc169715474ea237dd8b2beae440a5514686b8202506e12cb7dc1fca", 2024-09-10T03:42:04.736 INFO:teuthology.orchestra.run.smithi112.stdout: "in_progress": true, 2024-09-10T03:42:04.736 INFO:teuthology.orchestra.run.smithi112.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-09-10T03:42:04.737 INFO:teuthology.orchestra.run.smithi112.stdout: "services_complete": [], 2024-09-10T03:42:04.737 INFO:teuthology.orchestra.run.smithi112.stdout: "progress": "0/2 daemons upgraded", 2024-09-10T03:42:04.737 INFO:teuthology.orchestra.run.smithi112.stdout: "message": "Currently upgrading mgr daemons", 2024-09-10T03:42:04.737 INFO:teuthology.orchestra.run.smithi112.stdout: "is_paused": false 2024-09-10T03:42:04.737 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-10T03:42:04.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.736+0000 7f4c194c2640 1 -- 172.21.15.112:0/1416309841 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f4c0006de80 msgr2=0x7f4c00070340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:04.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.736+0000 7f4c194c2640 1 --2- 172.21.15.112:0/1416309841 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f4c0006de80 0x7f4c00070340 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc000c00 tx=0x7f4bfc005a20 comp rx=0 tx=0).stop 2024-09-10T03:42:04.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.736+0000 7f4c194c2640 1 -- 172.21.15.112:0/1416309841 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c14100840 msgr2=0x7f4c1406a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:04.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.736+0000 7f4c194c2640 1 --2- 172.21.15.112:0/1416309841 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c14100840 0x7f4c1406a620 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400c920 tx=0x7f4c0400cde0 comp rx=0 tx=0).stop 2024-09-10T03:42:04.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.736+0000 7f4c194c2640 1 -- 172.21.15.112:0/1416309841 shutdown_connections 2024-09-10T03:42:04.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.736+0000 7f4c194c2640 1 --2- 172.21.15.112:0/1416309841 >> [v2:172.21.15.112:6800/3600199018,v1:172.21.15.112:6801/3600199018] conn(0x7f4c0006de80 0x7f4c00070340 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:04.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.736+0000 7f4c194c2640 1 --2- 172.21.15.112:0/1416309841 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f4c14101a40 0x7f4c1419c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:04.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.736+0000 7f4c194c2640 1 --2- 172.21.15.112:0/1416309841 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c14100840 0x7f4c1406a620 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:04.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.736+0000 7f4c194c2640 1 -- 172.21.15.112:0/1416309841 >> 172.21.15.112:0/1416309841 conn(0x7f4c140fbff0 msgr2=0x7f4c14104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:42:04.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.737+0000 7f4c194c2640 1 -- 172.21.15.112:0/1416309841 shutdown_connections 2024-09-10T03:42:04.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:04.737+0000 7f4c194c2640 1 -- 172.21.15.112:0/1416309841 wait complete. 2024-09-10T03:42:05.010 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:04 smithi169 ceph-mon[35668]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:42:05.011 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:04 smithi169 ceph-mon[35668]: from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:42:05.011 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:04 smithi169 ceph-mon[35668]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:42:05.011 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:04 smithi169 ceph-mon[35668]: from='client.? 172.21.15.112:0/3008344484' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-10T03:42:05.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:04 smithi112 ceph-mon[27737]: from='client.14656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:42:05.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:04 smithi112 ceph-mon[27737]: from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:42:05.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:04 smithi112 ceph-mon[27737]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:42:05.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:04 smithi112 ceph-mon[27737]: from='client.? 172.21.15.112:0/3008344484' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-10T03:42:06.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:05 smithi169 ceph-mon[35668]: from='client.14672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:42:06.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:05 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:42:06.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:05 smithi169 ceph-mon[35668]: pgmap v228: 65 pgs: 65 active+clean; 45 MiB data, 305 MiB used, 536 GiB / 536 GiB avail; 3.9 MiB/s wr, 250 op/s 2024-09-10T03:42:06.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:05 smithi112 ceph-mon[27737]: from='client.14672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T03:42:06.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:05 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:42:06.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:05 smithi112 ceph-mon[27737]: pgmap v228: 65 pgs: 65 active+clean; 45 MiB data, 305 MiB used, 536 GiB / 536 GiB avail; 3.9 MiB/s wr, 250 op/s 2024-09-10T03:42:08.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:07 smithi169 ceph-mon[35668]: pgmap v229: 65 pgs: 65 active+clean; 55 MiB data, 379 MiB used, 536 GiB / 536 GiB avail; 4.7 MiB/s wr, 357 op/s 2024-09-10T03:42:08.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:07 smithi112 ceph-mon[27737]: pgmap v229: 65 pgs: 65 active+clean; 55 MiB data, 379 MiB used, 536 GiB / 536 GiB avail; 4.7 MiB/s wr, 357 op/s 2024-09-10T03:42:10.060 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:09 smithi169 ceph-mon[35668]: pgmap v230: 65 pgs: 65 active+clean; 60 MiB data, 404 MiB used, 536 GiB / 536 GiB avail; 5.2 MiB/s wr, 379 op/s 2024-09-10T03:42:10.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:09 smithi112 ceph-mon[27737]: pgmap v230: 65 pgs: 65 active+clean; 60 MiB data, 404 MiB used, 536 GiB / 536 GiB avail; 5.2 MiB/s wr, 379 op/s 2024-09-10T03:42:12.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:11 smithi169 ceph-mon[35668]: pgmap v231: 65 pgs: 65 active+clean; 77 MiB data, 462 MiB used, 536 GiB / 536 GiB avail; 6.7 MiB/s wr, 445 op/s 2024-09-10T03:42:12.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:11 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:12.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:11 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:12.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:11 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:42:12.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:11 smithi112 ceph-mon[27737]: pgmap v231: 65 pgs: 65 active+clean; 77 MiB data, 462 MiB used, 536 GiB / 536 GiB avail; 6.7 MiB/s wr, 445 op/s 2024-09-10T03:42:12.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:11 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:12.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:11 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:12.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:11 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:42:14.026 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:13 smithi169 ceph-mon[35668]: pgmap v232: 65 pgs: 65 active+clean; 77 MiB data, 474 MiB used, 536 GiB / 536 GiB avail; 4.7 MiB/s wr, 359 op/s 2024-09-10T03:42:14.315 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:13 smithi112 ceph-mon[27737]: pgmap v232: 65 pgs: 65 active+clean; 77 MiB data, 474 MiB used, 536 GiB / 536 GiB avail; 4.7 MiB/s wr, 359 op/s 2024-09-10T03:42:16.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:15 smithi169 ceph-mon[35668]: pgmap v233: 65 pgs: 65 active+clean; 83 MiB data, 480 MiB used, 536 GiB / 536 GiB avail; 4.6 MiB/s wr, 298 op/s 2024-09-10T03:42:16.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:15 smithi169 ceph-mon[35668]: Standby manager daemon smithi169.bkwgjl restarted 2024-09-10T03:42:16.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:15 smithi169 ceph-mon[35668]: Standby manager daemon smithi169.bkwgjl started 2024-09-10T03:42:16.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:15 smithi169 ceph-mon[35668]: from='mgr.? 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi169.bkwgjl/crt"}]: dispatch 2024-09-10T03:42:16.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:15 smithi169 ceph-mon[35668]: from='mgr.? 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T03:42:16.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:15 smithi169 ceph-mon[35668]: from='mgr.? 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi169.bkwgjl/key"}]: dispatch 2024-09-10T03:42:16.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:15 smithi169 ceph-mon[35668]: from='mgr.? 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T03:42:16.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:15 smithi112 ceph-mon[27737]: pgmap v233: 65 pgs: 65 active+clean; 83 MiB data, 480 MiB used, 536 GiB / 536 GiB avail; 4.6 MiB/s wr, 298 op/s 2024-09-10T03:42:16.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:15 smithi112 ceph-mon[27737]: Standby manager daemon smithi169.bkwgjl restarted 2024-09-10T03:42:16.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:15 smithi112 ceph-mon[27737]: Standby manager daemon smithi169.bkwgjl started 2024-09-10T03:42:16.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:15 smithi112 ceph-mon[27737]: from='mgr.? 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi169.bkwgjl/crt"}]: dispatch 2024-09-10T03:42:16.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:15 smithi112 ceph-mon[27737]: from='mgr.? 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T03:42:16.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:15 smithi112 ceph-mon[27737]: from='mgr.? 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi169.bkwgjl/key"}]: dispatch 2024-09-10T03:42:16.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:15 smithi112 ceph-mon[27737]: from='mgr.? 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T03:42:17.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:16 smithi169 ceph-mon[35668]: mgrmap e20: smithi112.pqrlrb(active, since 6m), standbys: smithi169.bkwgjl 2024-09-10T03:42:17.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:16 smithi112 ceph-mon[27737]: mgrmap e20: smithi112.pqrlrb(active, since 6m), standbys: smithi169.bkwgjl 2024-09-10T03:42:18.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:17 smithi169 ceph-mon[35668]: pgmap v234: 65 pgs: 65 active+clean; 86 MiB data, 491 MiB used, 536 GiB / 536 GiB avail; 3.6 MiB/s wr, 218 op/s 2024-09-10T03:42:18.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:17 smithi112 ceph-mon[27737]: pgmap v234: 65 pgs: 65 active+clean; 86 MiB data, 491 MiB used, 536 GiB / 536 GiB avail; 3.6 MiB/s wr, 218 op/s 2024-09-10T03:42:20.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:19 smithi169 ceph-mon[35668]: pgmap v235: 65 pgs: 65 active+clean; 99 MiB data, 561 MiB used, 536 GiB / 536 GiB avail; 3.9 MiB/s wr, 272 op/s 2024-09-10T03:42:20.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:19 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:42:20.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:19 smithi112 ceph-mon[27737]: pgmap v235: 65 pgs: 65 active+clean; 99 MiB data, 561 MiB used, 536 GiB / 536 GiB avail; 3.9 MiB/s wr, 272 op/s 2024-09-10T03:42:20.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:19 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:42:22.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:21 smithi169 ceph-mon[35668]: pgmap v236: 65 pgs: 65 active+clean; 139 MiB data, 727 MiB used, 536 GiB / 536 GiB avail; 7.1 MiB/s wr, 822 op/s 2024-09-10T03:42:22.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:21 smithi112 ceph-mon[27737]: pgmap v236: 65 pgs: 65 active+clean; 139 MiB data, 727 MiB used, 536 GiB / 536 GiB avail; 7.1 MiB/s wr, 822 op/s 2024-09-10T03:42:24.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:23 smithi169 ceph-mon[35668]: pgmap v237: 65 pgs: 65 active+clean; 147 MiB data, 754 MiB used, 536 GiB / 536 GiB avail; 6.3 MiB/s wr, 819 op/s 2024-09-10T03:42:24.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:23 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:24.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:23 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:24.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:23 smithi112 ceph-mon[27737]: pgmap v237: 65 pgs: 65 active+clean; 147 MiB data, 754 MiB used, 536 GiB / 536 GiB avail; 6.3 MiB/s wr, 819 op/s 2024-09-10T03:42:24.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:23 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:24.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:23 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:25.194 INFO:tasks.workunit.client.1.smithi169.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-09-10T03:42:25.194 INFO:tasks.workunit.client.1.smithi169.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-09-10T03:42:25.194 INFO:tasks.workunit.client.1.smithi169.stderr:+ make 2024-09-10T03:42:25.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:25 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:25.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:25 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:25.526 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:25 smithi169 ceph-mon[35668]: pgmap v238: 65 pgs: 65 active+clean; 151 MiB data, 778 MiB used, 536 GiB / 536 GiB avail; 6.6 MiB/s wr, 826 op/s 2024-09-10T03:42:25.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:25 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:25.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:25 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:25.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:25 smithi112 ceph-mon[27737]: pgmap v238: 65 pgs: 65 active+clean; 151 MiB data, 778 MiB used, 536 GiB / 536 GiB avail; 6.6 MiB/s wr, 826 op/s 2024-09-10T03:42:25.946 INFO:tasks.workunit.client.1.smithi169.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-09-10T03:42:26.493 INFO:tasks.workunit.client.1.smithi169.stderr:++ readlink -f fsstress 2024-09-10T03:42:26.493 INFO:tasks.workunit.client.1.smithi169.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-09-10T03:42:26.494 INFO:tasks.workunit.client.1.smithi169.stderr:+ popd 2024-09-10T03:42:26.494 INFO:tasks.workunit.client.1.smithi169.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-09-10T03:42:26.494 INFO:tasks.workunit.client.1.smithi169.stderr:+ popd 2024-09-10T03:42:26.494 INFO:tasks.workunit.client.1.smithi169.stdout:~/cephtest/mnt.1/client.1/tmp 2024-09-10T03:42:26.495 INFO:tasks.workunit.client.1.smithi169.stderr:++ mktemp -d -p . 2024-09-10T03:42:26.495 INFO:tasks.workunit.client.1.smithi169.stderr:+ T=./tmp.B2eVL6iJL2 2024-09-10T03:42:26.496 INFO:tasks.workunit.client.1.smithi169.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.B2eVL6iJL2 -l 1 -n 1000 -p 10 -v 2024-09-10T03:42:26.497 INFO:tasks.workunit.client.1.smithi169.stdout:seed = 1725454580 2024-09-10T03:42:26.500 INFO:tasks.workunit.client.1.smithi169.stdout:0/0: dwrite - no filename 2024-09-10T03:42:26.500 INFO:tasks.workunit.client.1.smithi169.stdout:0/1: dread - no filename 2024-09-10T03:42:26.501 INFO:tasks.workunit.client.1.smithi169.stdout:1/0: rmdir - no directory 2024-09-10T03:42:26.501 INFO:tasks.workunit.client.1.smithi169.stdout:0/2: creat f0 x:0 0 0 2024-09-10T03:42:26.502 INFO:tasks.workunit.client.1.smithi169.stdout:2/0: readlink - no filename 2024-09-10T03:42:26.503 INFO:tasks.workunit.client.1.smithi169.stdout:1/1: creat f0 x:0 0 0 2024-09-10T03:42:26.503 INFO:tasks.workunit.client.1.smithi169.stdout:1/2: write f0 [321721,13662] 0 2024-09-10T03:42:26.503 INFO:tasks.workunit.client.1.smithi169.stdout:1/3: stat f0 0 2024-09-10T03:42:26.504 INFO:tasks.workunit.client.1.smithi169.stdout:0/3: symlink l1 0 2024-09-10T03:42:26.506 INFO:tasks.workunit.client.1.smithi169.stdout:2/1: mkdir d0 0 2024-09-10T03:42:26.507 INFO:tasks.workunit.client.1.smithi169.stdout:4/0: dread - no filename 2024-09-10T03:42:26.507 INFO:tasks.workunit.client.1.smithi169.stdout:4/1: rename - no filename 2024-09-10T03:42:26.508 INFO:tasks.workunit.client.1.smithi169.stdout:4/2: write - no filename 2024-09-10T03:42:26.508 INFO:tasks.workunit.client.1.smithi169.stdout:4/3: dread - no filename 2024-09-10T03:42:26.508 INFO:tasks.workunit.client.1.smithi169.stdout:4/4: chown . 260129 1 2024-09-10T03:42:26.508 INFO:tasks.workunit.client.1.smithi169.stdout:4/5: truncate - no filename 2024-09-10T03:42:26.508 INFO:tasks.workunit.client.1.smithi169.stdout:4/6: chown . 30715862 1 2024-09-10T03:42:26.508 INFO:tasks.workunit.client.1.smithi169.stdout:4/7: dread - no filename 2024-09-10T03:42:26.508 INFO:tasks.workunit.client.1.smithi169.stdout:3/0: mkdir d0 0 2024-09-10T03:42:26.509 INFO:tasks.workunit.client.1.smithi169.stdout:2/2: mkdir d0/d1 0 2024-09-10T03:42:26.512 INFO:tasks.workunit.client.1.smithi169.stdout:5/0: fsync - no filename 2024-09-10T03:42:26.515 INFO:tasks.workunit.client.1.smithi169.stdout:4/8: mkdir d0 0 2024-09-10T03:42:26.532 INFO:tasks.workunit.client.1.smithi169.stdout:3/1: mknod d0/c1 0 2024-09-10T03:42:26.533 INFO:tasks.workunit.client.1.smithi169.stdout:2/3: rmdir d0/d1 0 2024-09-10T03:42:26.533 INFO:tasks.workunit.client.1.smithi169.stdout:2/4: dread - no filename 2024-09-10T03:42:26.534 INFO:tasks.workunit.client.1.smithi169.stdout:5/1: creat f0 x:0 0 0 2024-09-10T03:42:26.537 INFO:tasks.workunit.client.1.smithi169.stdout:0/4: fsync f0 0 2024-09-10T03:42:26.540 INFO:tasks.workunit.client.1.smithi169.stdout:2/5: creat d0/f2 x:0 0 0 2024-09-10T03:42:26.543 INFO:tasks.workunit.client.1.smithi169.stdout:0/5: getdents . 0 2024-09-10T03:42:26.543 INFO:tasks.workunit.client.1.smithi169.stdout:0/6: dread - f0 zero size 2024-09-10T03:42:26.543 INFO:tasks.workunit.client.1.smithi169.stdout:0/7: write f0 [367523,87495] 0 2024-09-10T03:42:26.545 INFO:tasks.workunit.client.1.smithi169.stdout:2/6: creat d0/f3 x:0 0 0 2024-09-10T03:42:26.550 INFO:tasks.workunit.client.1.smithi169.stdout:0/8: symlink l2 0 2024-09-10T03:42:26.553 INFO:tasks.workunit.client.1.smithi169.stdout:2/7: creat d0/f4 x:0 0 0 2024-09-10T03:42:26.554 INFO:tasks.workunit.client.1.smithi169.stdout:2/8: chown d0/f4 6349016 1 2024-09-10T03:42:26.555 INFO:tasks.workunit.client.1.smithi169.stdout:2/9: mkdir d0/d5 0 2024-09-10T03:42:26.559 INFO:tasks.workunit.client.1.smithi169.stdout:2/10: link d0/f3 d0/d5/f6 0 2024-09-10T03:42:26.559 INFO:tasks.workunit.client.1.smithi169.stdout:2/11: dread - d0/f4 zero size 2024-09-10T03:42:26.562 INFO:tasks.workunit.client.1.smithi169.stdout:2/12: rmdir d0/d5 39 2024-09-10T03:42:26.566 INFO:tasks.workunit.client.1.smithi169.stdout:2/13: link d0/d5/f6 d0/d5/f7 0 2024-09-10T03:42:26.571 INFO:tasks.workunit.client.1.smithi169.stdout:2/14: unlink d0/f4 0 2024-09-10T03:42:26.678 INFO:tasks.workunit.client.1.smithi169.stdout:5/2: fsync f0 0 2024-09-10T03:42:26.703 INFO:tasks.workunit.client.1.smithi169.stdout:0/9: getdents . 0 2024-09-10T03:42:26.705 INFO:tasks.workunit.client.1.smithi169.stdout:1/4: truncate f0 40009 0 2024-09-10T03:42:26.709 INFO:tasks.workunit.client.1.smithi169.stdout:2/15: dwrite d0/d5/f7 [0,4194304] 0 2024-09-10T03:42:26.709 INFO:tasks.workunit.client.1.smithi169.stdout:2/16: dread - d0/f2 zero size 2024-09-10T03:42:26.713 INFO:tasks.workunit.client.1.smithi169.stdout:3/2: write - no filename 2024-09-10T03:42:26.713 INFO:tasks.workunit.client.1.smithi169.stdout:7/0: link - no file 2024-09-10T03:42:26.714 INFO:tasks.workunit.client.1.smithi169.stdout:7/1: read - no filename 2024-09-10T03:42:26.714 INFO:tasks.workunit.client.1.smithi169.stdout:9/0: dread - no filename 2024-09-10T03:42:26.714 INFO:tasks.workunit.client.1.smithi169.stdout:7/2: rmdir - no directory 2024-09-10T03:42:26.714 INFO:tasks.workunit.client.1.smithi169.stdout:9/1: dread - no filename 2024-09-10T03:42:26.714 INFO:tasks.workunit.client.1.smithi169.stdout:9/2: dread - no filename 2024-09-10T03:42:26.714 INFO:tasks.workunit.client.1.smithi169.stdout:9/3: dread - no filename 2024-09-10T03:42:26.715 INFO:tasks.workunit.client.1.smithi169.stdout:9/4: truncate - no filename 2024-09-10T03:42:26.715 INFO:tasks.workunit.client.1.smithi169.stdout:9/5: dread - no filename 2024-09-10T03:42:26.715 INFO:tasks.workunit.client.1.smithi169.stdout:9/6: rmdir - no directory 2024-09-10T03:42:26.715 INFO:tasks.workunit.client.1.smithi169.stdout:9/7: dwrite - no filename 2024-09-10T03:42:26.715 INFO:tasks.workunit.client.1.smithi169.stdout:9/8: link - no file 2024-09-10T03:42:26.715 INFO:tasks.workunit.client.1.smithi169.stdout:7/3: chown . 1134548 1 2024-09-10T03:42:26.715 INFO:tasks.workunit.client.1.smithi169.stdout:7/4: dread - no filename 2024-09-10T03:42:26.716 INFO:tasks.workunit.client.1.smithi169.stdout:7/5: dwrite - no filename 2024-09-10T03:42:26.716 INFO:tasks.workunit.client.1.smithi169.stdout:9/9: chown . 878893 1 2024-09-10T03:42:26.716 INFO:tasks.workunit.client.1.smithi169.stdout:6/0: rename - no filename 2024-09-10T03:42:26.716 INFO:tasks.workunit.client.1.smithi169.stdout:6/1: link - no file 2024-09-10T03:42:26.716 INFO:tasks.workunit.client.1.smithi169.stdout:6/2: unlink - no file 2024-09-10T03:42:26.716 INFO:tasks.workunit.client.1.smithi169.stdout:8/0: read - no filename 2024-09-10T03:42:26.742 INFO:tasks.workunit.client.1.smithi169.stdout:0/10: fdatasync f0 0 2024-09-10T03:42:26.742 INFO:tasks.workunit.client.1.smithi169.stdout:0/11: chown l1 244 1 2024-09-10T03:42:26.742 INFO:tasks.workunit.client.1.smithi169.stdout:0/12: readlink l1 0 2024-09-10T03:42:26.744 INFO:tasks.workunit.client.1.smithi169.stdout:0/13: dread f0 [0,4194304] 0 2024-09-10T03:42:26.744 INFO:tasks.workunit.client.1.smithi169.stdout:0/14: truncate f0 911585 0 2024-09-10T03:42:26.810 INFO:tasks.workunit.client.1.smithi169.stdout:1/5: fsync f0 0 2024-09-10T03:42:27.264 INFO:tasks.workunit.client.1.smithi169.stdout:2/17: mknod d0/d5/c8 0 2024-09-10T03:42:27.271 INFO:tasks.workunit.client.1.smithi169.stdout:2/18: write d0/d5/f7 [3534200,114331] 0 2024-09-10T03:42:27.271 INFO:tasks.workunit.client.1.smithi169.stdout:2/19: dread - d0/f2 zero size 2024-09-10T03:42:27.336 INFO:tasks.workunit.client.1.smithi169.stdout:3/3: creat d0/f2 x:0 0 0 2024-09-10T03:42:27.337 INFO:tasks.workunit.client.1.smithi169.stdout:7/6: mkdir d0 0 2024-09-10T03:42:27.338 INFO:tasks.workunit.client.1.smithi169.stdout:9/10: mknod c0 0 2024-09-10T03:42:27.339 INFO:tasks.workunit.client.1.smithi169.stdout:9/11: rmdir - no directory 2024-09-10T03:42:27.339 INFO:tasks.workunit.client.1.smithi169.stdout:9/12: dread - no filename 2024-09-10T03:42:27.343 INFO:tasks.workunit.client.1.smithi169.stdout:5/3: rename f0 to f1 0 2024-09-10T03:42:27.345 INFO:tasks.workunit.client.1.smithi169.stdout:6/3: mkdir d0 0 2024-09-10T03:42:27.458 INFO:tasks.workunit.client.1.smithi169.stdout:0/15: mknod c3 0 2024-09-10T03:42:27.458 INFO:tasks.workunit.client.1.smithi169.stdout:0/16: truncate f0 1899266 0 2024-09-10T03:42:27.459 INFO:tasks.workunit.client.1.smithi169.stdout:1/6: rename f0 to f1 0 2024-09-10T03:42:27.459 INFO:tasks.workunit.client.1.smithi169.stdout:1/7: write f1 [421391,128206] 0 2024-09-10T03:42:28.149 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:27 smithi169 ceph-mon[35668]: pgmap v239: 65 pgs: 65 active+clean; 160 MiB data, 801 MiB used, 536 GiB / 536 GiB avail; 6.9 MiB/s wr, 835 op/s 2024-09-10T03:42:28.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:27 smithi112 ceph-mon[27737]: pgmap v239: 65 pgs: 65 active+clean; 160 MiB data, 801 MiB used, 536 GiB / 536 GiB avail; 6.9 MiB/s wr, 835 op/s 2024-09-10T03:42:28.913 INFO:tasks.workunit.client.1.smithi169.stdout:2/20: mknod d0/d5/c9 0 2024-09-10T03:42:28.913 INFO:tasks.workunit.client.1.smithi169.stdout:3/4: symlink d0/l3 0 2024-09-10T03:42:28.913 INFO:tasks.workunit.client.1.smithi169.stdout:9/13: mknod c1 0 2024-09-10T03:42:28.913 INFO:tasks.workunit.client.1.smithi169.stdout:9/14: dwrite - no filename 2024-09-10T03:42:28.914 INFO:tasks.workunit.client.1.smithi169.stdout:5/4: rename f1 to f2 0 2024-09-10T03:42:28.915 INFO:tasks.workunit.client.1.smithi169.stdout:5/5: chown f2 457 1 2024-09-10T03:42:28.915 INFO:tasks.workunit.client.1.smithi169.stdout:5/6: dread - f2 zero size 2024-09-10T03:42:28.915 INFO:tasks.workunit.client.1.smithi169.stdout:5/7: write f2 [850846,115358] 0 2024-09-10T03:42:28.915 INFO:tasks.workunit.client.1.smithi169.stdout:6/4: mknod d0/c1 0 2024-09-10T03:42:28.915 INFO:tasks.workunit.client.1.smithi169.stdout:5/8: write f2 [488908,87380] 0 2024-09-10T03:42:28.915 INFO:tasks.workunit.client.1.smithi169.stdout:5/9: stat f2 0 2024-09-10T03:42:28.916 INFO:tasks.workunit.client.1.smithi169.stdout:0/17: link f0 f4 0 2024-09-10T03:42:28.916 INFO:tasks.workunit.client.1.smithi169.stdout:0/18: write f0 [2003128,124488] 0 2024-09-10T03:42:28.917 INFO:tasks.workunit.client.1.smithi169.stdout:2/21: unlink d0/f3 0 2024-09-10T03:42:28.918 INFO:tasks.workunit.client.1.smithi169.stdout:9/15: rename c1 to c2 0 2024-09-10T03:42:28.919 INFO:tasks.workunit.client.1.smithi169.stdout:6/5: symlink d0/l2 0 2024-09-10T03:42:28.919 INFO:tasks.workunit.client.1.smithi169.stdout:5/10: mkdir d3 0 2024-09-10T03:42:28.919 INFO:tasks.workunit.client.1.smithi169.stdout:5/11: read f2 [572476,26171] 0 2024-09-10T03:42:28.920 INFO:tasks.workunit.client.1.smithi169.stdout:0/19: mkdir d5 0 2024-09-10T03:42:28.920 INFO:tasks.workunit.client.1.smithi169.stdout:0/20: chown f4 4129962 1 2024-09-10T03:42:28.923 INFO:tasks.workunit.client.1.smithi169.stdout:6/6: creat d0/f3 x:0 0 0 2024-09-10T03:42:28.924 INFO:tasks.workunit.client.1.smithi169.stdout:0/21: creat d5/f6 x:0 0 0 2024-09-10T03:42:28.927 INFO:tasks.workunit.client.1.smithi169.stdout:2/22: dread d0/d5/f6 [0,4194304] 0 2024-09-10T03:42:28.927 INFO:tasks.workunit.client.1.smithi169.stdout:2/23: readlink - no filename 2024-09-10T03:42:28.927 INFO:tasks.workunit.client.1.smithi169.stdout:2/24: stat d0/d5/f7 0 2024-09-10T03:42:28.927 INFO:tasks.workunit.client.1.smithi169.stdout:6/7: mknod d0/c4 0 2024-09-10T03:42:28.930 INFO:tasks.workunit.client.1.smithi169.stdout:6/8: link d0/l2 d0/l5 0 2024-09-10T03:42:28.933 INFO:tasks.workunit.client.1.smithi169.stdout:6/9: mkdir d0/d6 0 2024-09-10T03:42:28.939 INFO:tasks.workunit.client.1.smithi169.stdout:0/22: dread f4 [0,4194304] 0 2024-09-10T03:42:28.940 INFO:tasks.workunit.client.1.smithi169.stdout:3/5: dwrite d0/f2 [0,4194304] 0 2024-09-10T03:42:28.941 INFO:tasks.workunit.client.1.smithi169.stdout:0/23: creat d5/f7 x:0 0 0 2024-09-10T03:42:28.941 INFO:tasks.workunit.client.1.smithi169.stdout:0/24: truncate f0 2308266 0 2024-09-10T03:42:28.946 INFO:tasks.workunit.client.1.smithi169.stdout:3/6: creat d0/f4 x:0 0 0 2024-09-10T03:42:28.948 INFO:tasks.workunit.client.1.smithi169.stdout:0/25: rename f0 to d5/f8 0 2024-09-10T03:42:28.948 INFO:tasks.workunit.client.1.smithi169.stdout:0/26: write d5/f6 [684778,51692] 0 2024-09-10T03:42:28.950 INFO:tasks.workunit.client.1.smithi169.stdout:3/7: symlink d0/l5 0 2024-09-10T03:42:28.951 INFO:tasks.workunit.client.1.smithi169.stdout:5/12: dwrite f2 [0,4194304] 0 2024-09-10T03:42:28.952 INFO:tasks.workunit.client.1.smithi169.stdout:3/8: creat d0/f6 x:0 0 0 2024-09-10T03:42:28.956 INFO:tasks.workunit.client.1.smithi169.stdout:3/9: creat d0/f7 x:0 0 0 2024-09-10T03:42:28.963 INFO:tasks.workunit.client.1.smithi169.stdout:5/13: dread f2 [0,4194304] 0 2024-09-10T03:42:28.964 INFO:tasks.workunit.client.1.smithi169.stdout:6/10: dwrite d0/f3 [0,4194304] 0 2024-09-10T03:42:28.965 INFO:tasks.workunit.client.1.smithi169.stdout:6/11: truncate d0/f3 5102640 0 2024-09-10T03:42:28.965 INFO:tasks.workunit.client.1.smithi169.stdout:5/14: mknod d3/c4 0 2024-09-10T03:42:28.967 INFO:tasks.workunit.client.1.smithi169.stdout:0/27: dwrite f4 [0,4194304] 0 2024-09-10T03:42:28.967 INFO:tasks.workunit.client.1.smithi169.stdout:0/28: readlink l2 0 2024-09-10T03:42:28.972 INFO:tasks.workunit.client.1.smithi169.stdout:5/15: dread f2 [0,4194304] 0 2024-09-10T03:42:28.977 INFO:tasks.workunit.client.1.smithi169.stdout:3/10: dread d0/f2 [0,4194304] 0 2024-09-10T03:42:28.987 INFO:tasks.workunit.client.1.smithi169.stdout:3/11: dread d0/f2 [0,4194304] 0 2024-09-10T03:42:28.989 INFO:tasks.workunit.client.1.smithi169.stdout:3/12: rename d0/l3 to d0/l8 0 2024-09-10T03:42:28.991 INFO:tasks.workunit.client.1.smithi169.stdout:3/13: mknod d0/c9 0 2024-09-10T03:42:28.993 INFO:tasks.workunit.client.1.smithi169.stdout:3/14: rmdir d0 39 2024-09-10T03:42:28.996 INFO:tasks.workunit.client.1.smithi169.stdout:3/15: unlink d0/f6 0 2024-09-10T03:42:28.997 INFO:tasks.workunit.client.1.smithi169.stdout:6/12: dwrite d0/f3 [0,4194304] 0 2024-09-10T03:42:29.000 INFO:tasks.workunit.client.1.smithi169.stdout:5/16: dwrite f2 [0,4194304] 0 2024-09-10T03:42:29.000 INFO:tasks.workunit.client.1.smithi169.stdout:6/13: rename d0/l5 to d0/d6/l7 0 2024-09-10T03:42:29.001 INFO:tasks.workunit.client.1.smithi169.stdout:3/16: write d0/f2 [672948,107914] 0 2024-09-10T03:42:29.001 INFO:tasks.workunit.client.1.smithi169.stdout:3/17: truncate d0/f7 53174 0 2024-09-10T03:42:29.002 INFO:tasks.workunit.client.1.smithi169.stdout:3/18: dread d0/f7 [0,4194304] 0 2024-09-10T03:42:29.002 INFO:tasks.workunit.client.1.smithi169.stdout:3/19: truncate d0/f2 4790840 0 2024-09-10T03:42:29.004 INFO:tasks.workunit.client.1.smithi169.stdout:5/17: creat d3/f5 x:0 0 0 2024-09-10T03:42:29.009 INFO:tasks.workunit.client.1.smithi169.stdout:6/14: dread d0/f3 [0,4194304] 0 2024-09-10T03:42:29.015 INFO:tasks.workunit.client.1.smithi169.stdout:5/18: dread f2 [0,4194304] 0 2024-09-10T03:42:29.028 INFO:tasks.workunit.client.1.smithi169.stdout:3/20: dwrite d0/f2 [0,4194304] 0 2024-09-10T03:42:29.043 INFO:tasks.workunit.client.1.smithi169.stdout:3/21: dwrite d0/f4 [0,4194304] 0 2024-09-10T03:42:29.047 INFO:tasks.workunit.client.1.smithi169.stdout:3/22: creat d0/fa x:0 0 0 2024-09-10T03:42:29.047 INFO:tasks.workunit.client.1.smithi169.stdout:3/23: write d0/fa [593709,128204] 0 2024-09-10T03:42:29.050 INFO:tasks.workunit.client.1.smithi169.stdout:3/24: link d0/c9 d0/cb 0 2024-09-10T03:42:29.050 INFO:tasks.workunit.client.1.smithi169.stdout:3/25: fdatasync d0/f4 0 2024-09-10T03:42:29.050 INFO:tasks.workunit.client.1.smithi169.stdout:3/26: write d0/fa [1052246,50382] 0 2024-09-10T03:42:29.276 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:29.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:29.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:42:29.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:42:29.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:29.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:42:29.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr fail", "who": "smithi112.pqrlrb"}]: dispatch 2024-09-10T03:42:29.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: Activating manager daemon smithi169.bkwgjl 2024-09-10T03:42:29.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "mgr fail", "who": "smithi112.pqrlrb"}]': finished 2024-09-10T03:42:29.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: osdmap e46: 6 total, 6 up, 6 in 2024-09-10T03:42:29.277 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: mgrmap e21: smithi169.bkwgjl(active, starting, since 0.00990586s) 2024-09-10T03:42:29.278 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2024-09-10T03:42:29.278 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:42:29.278 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.uuenbo"}]: dispatch 2024-09-10T03:42:29.278 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.ifnhnl"}]: dispatch 2024-09-10T03:42:29.278 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi169.wfmonc"}]: dispatch 2024-09-10T03:42:29.278 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi169.gcmhul"}]: dispatch 2024-09-10T03:42:29.278 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mgr metadata", "who": "smithi169.bkwgjl", "id": "smithi169.bkwgjl"}]: dispatch 2024-09-10T03:42:29.278 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T03:42:29.278 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T03:42:29.278 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T03:42:29.278 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T03:42:29.278 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T03:42:29.278 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T03:42:29.279 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T03:42:29.279 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T03:42:29.279 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T03:42:29.279 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:29 smithi169 ceph-mon[35668]: Manager daemon smithi169.bkwgjl is now available 2024-09-10T03:42:29.551 INFO:tasks.workunit.client.1.smithi169.stdout:9/16: rename c2 to c3 0 2024-09-10T03:42:29.551 INFO:tasks.workunit.client.1.smithi169.stdout:9/17: write - no filename 2024-09-10T03:42:29.553 INFO:tasks.workunit.client.1.smithi169.stdout:9/18: symlink l4 0 2024-09-10T03:42:29.555 INFO:tasks.workunit.client.1.smithi169.stdout:6/15: fsync d0/f3 0 2024-09-10T03:42:29.568 INFO:tasks.workunit.client.1.smithi169.stdout:0/29: link d5/f8 d5/f9 0 2024-09-10T03:42:29.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:29.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:29.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T03:42:29.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T03:42:29.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' 2024-09-10T03:42:29.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:42:29.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd=[{"prefix": "mgr fail", "who": "smithi112.pqrlrb"}]: dispatch 2024-09-10T03:42:29.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: Activating manager daemon smithi169.bkwgjl 2024-09-10T03:42:29.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.14221 172.21.15.112:0/1723659225' entity='mgr.smithi112.pqrlrb' cmd='[{"prefix": "mgr fail", "who": "smithi112.pqrlrb"}]': finished 2024-09-10T03:42:29.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: osdmap e46: 6 total, 6 up, 6 in 2024-09-10T03:42:29.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: mgrmap e21: smithi169.bkwgjl(active, starting, since 0.00990586s) 2024-09-10T03:42:29.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mon metadata", "id": "smithi112"}]: dispatch 2024-09-10T03:42:29.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2024-09-10T03:42:29.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.uuenbo"}]: dispatch 2024-09-10T03:42:29.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi112.ifnhnl"}]: dispatch 2024-09-10T03:42:29.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi169.wfmonc"}]: dispatch 2024-09-10T03:42:29.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi169.gcmhul"}]: dispatch 2024-09-10T03:42:29.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mgr metadata", "who": "smithi169.bkwgjl", "id": "smithi169.bkwgjl"}]: dispatch 2024-09-10T03:42:29.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T03:42:29.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T03:42:29.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T03:42:29.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T03:42:29.583 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T03:42:29.583 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T03:42:29.583 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T03:42:29.583 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T03:42:29.583 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T03:42:29.583 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:29 smithi112 ceph-mon[27737]: Manager daemon smithi169.bkwgjl is now available 2024-09-10T03:42:29.616 INFO:tasks.workunit.client.1.smithi169.stdout:8/1: sync 2024-09-10T03:42:29.616 INFO:tasks.workunit.client.1.smithi169.stdout:4/9: sync 2024-09-10T03:42:29.617 INFO:tasks.workunit.client.1.smithi169.stdout:4/10: fdatasync - no filename 2024-09-10T03:42:29.617 INFO:tasks.workunit.client.1.smithi169.stdout:4/11: write - no filename 2024-09-10T03:42:29.617 INFO:tasks.workunit.client.1.smithi169.stdout:4/12: read - no filename 2024-09-10T03:42:29.617 INFO:tasks.workunit.client.1.smithi169.stdout:4/13: write - no filename 2024-09-10T03:42:29.617 INFO:tasks.workunit.client.1.smithi169.stdout:8/2: getdents . 0 2024-09-10T03:42:29.617 INFO:tasks.workunit.client.1.smithi169.stdout:7/7: sync 2024-09-10T03:42:29.617 INFO:tasks.workunit.client.1.smithi169.stdout:7/8: read - no filename 2024-09-10T03:42:29.617 INFO:tasks.workunit.client.1.smithi169.stdout:7/9: write - no filename 2024-09-10T03:42:29.617 INFO:tasks.workunit.client.1.smithi169.stdout:1/8: sync 2024-09-10T03:42:29.617 INFO:tasks.workunit.client.1.smithi169.stdout:1/9: fsync f1 0 2024-09-10T03:42:29.617 INFO:tasks.workunit.client.1.smithi169.stdout:1/10: chown f1 14440316 1 2024-09-10T03:42:29.617 INFO:tasks.workunit.client.1.smithi169.stdout:1/11: stat f1 0 2024-09-10T03:42:29.617 INFO:tasks.workunit.client.1.smithi169.stdout:1/12: readlink - no filename 2024-09-10T03:42:29.617 INFO:tasks.workunit.client.1.smithi169.stdout:1/13: fsync f1 0 2024-09-10T03:42:29.618 INFO:tasks.workunit.client.1.smithi169.stdout:4/14: creat d0/f1 x:0 0 0 2024-09-10T03:42:29.618 INFO:tasks.workunit.client.1.smithi169.stdout:4/15: write d0/f1 [133804,86963] 0 2024-09-10T03:42:29.618 INFO:tasks.workunit.client.1.smithi169.stdout:8/3: symlink l0 0 2024-09-10T03:42:29.618 INFO:tasks.workunit.client.1.smithi169.stdout:7/10: creat d0/f1 x:0 0 0 2024-09-10T03:42:29.619 INFO:tasks.workunit.client.1.smithi169.stdout:7/11: write d0/f1 [155482,4701] 0 2024-09-10T03:42:29.620 INFO:tasks.workunit.client.1.smithi169.stdout:4/16: mkdir d0/d2 0 2024-09-10T03:42:29.622 INFO:tasks.workunit.client.1.smithi169.stdout:0/30: dwrite f4 [0,4194304] 0 2024-09-10T03:42:29.622 INFO:tasks.workunit.client.1.smithi169.stdout:0/31: write d5/f7 [129233,44827] 0 2024-09-10T03:42:29.622 INFO:tasks.workunit.client.1.smithi169.stdout:4/17: write d0/f1 [92426,97093] 0 2024-09-10T03:42:29.622 INFO:tasks.workunit.client.1.smithi169.stdout:8/4: unlink l0 0 2024-09-10T03:42:29.622 INFO:tasks.workunit.client.1.smithi169.stdout:8/5: fdatasync - no filename 2024-09-10T03:42:29.622 INFO:tasks.workunit.client.1.smithi169.stdout:8/6: write - no filename 2024-09-10T03:42:29.622 INFO:tasks.workunit.client.1.smithi169.stdout:8/7: truncate - no filename 2024-09-10T03:42:29.623 INFO:tasks.workunit.client.1.smithi169.stdout:7/12: unlink d0/f1 0 2024-09-10T03:42:29.623 INFO:tasks.workunit.client.1.smithi169.stdout:7/13: dwrite - no filename 2024-09-10T03:42:29.628 INFO:tasks.workunit.client.1.smithi169.stdout:0/32: rename d5/f8 to d5/fa 0 2024-09-10T03:42:29.628 INFO:tasks.workunit.client.1.smithi169.stdout:0/33: chown d5/f7 192 1 2024-09-10T03:42:29.629 INFO:tasks.workunit.client.1.smithi169.stdout:4/18: mknod d0/d2/c3 0 2024-09-10T03:42:29.630 INFO:tasks.workunit.client.1.smithi169.stdout:8/8: mkdir d1 0 2024-09-10T03:42:29.630 INFO:tasks.workunit.client.1.smithi169.stdout:8/9: chown d1 6 1 2024-09-10T03:42:29.631 INFO:tasks.workunit.client.1.smithi169.stdout:8/10: dread - no filename 2024-09-10T03:42:29.631 INFO:tasks.workunit.client.1.smithi169.stdout:7/14: creat d0/f2 x:0 0 0 2024-09-10T03:42:29.631 INFO:tasks.workunit.client.1.smithi169.stdout:7/15: write d0/f2 [590135,15890] 0 2024-09-10T03:42:29.633 INFO:tasks.workunit.client.1.smithi169.stdout:4/19: mknod d0/c4 0 2024-09-10T03:42:29.635 INFO:tasks.workunit.client.1.smithi169.stdout:8/11: rmdir d1 0 2024-09-10T03:42:29.635 INFO:tasks.workunit.client.1.smithi169.stdout:8/12: write - no filename 2024-09-10T03:42:29.637 INFO:tasks.workunit.client.1.smithi169.stdout:4/20: creat d0/d2/f5 x:0 0 0 2024-09-10T03:42:29.639 INFO:tasks.workunit.client.1.smithi169.stdout:8/13: creat f2 x:0 0 0 2024-09-10T03:42:29.640 INFO:tasks.workunit.client.1.smithi169.stdout:8/14: dread - f2 zero size 2024-09-10T03:42:29.640 INFO:tasks.workunit.client.1.smithi169.stdout:8/15: fdatasync f2 0 2024-09-10T03:42:29.641 INFO:tasks.workunit.client.1.smithi169.stdout:5/19: truncate f2 2694060 0 2024-09-10T03:42:29.641 INFO:tasks.workunit.client.1.smithi169.stdout:4/21: mkdir d0/d6 0 2024-09-10T03:42:29.643 INFO:tasks.workunit.client.1.smithi169.stdout:8/16: mkdir d3 0 2024-09-10T03:42:29.643 INFO:tasks.workunit.client.1.smithi169.stdout:8/17: truncate f2 733071 0 2024-09-10T03:42:29.644 INFO:tasks.workunit.client.1.smithi169.stdout:8/18: write f2 [753038,35247] 0 2024-09-10T03:42:29.644 INFO:tasks.workunit.client.1.smithi169.stdout:0/34: truncate d5/f6 498976 0 2024-09-10T03:42:29.644 INFO:tasks.workunit.client.1.smithi169.stdout:4/22: mkdir d0/d6/d7 0 2024-09-10T03:42:29.647 INFO:tasks.workunit.client.1.smithi169.stdout:0/35: mkdir d5/db 0 2024-09-10T03:42:29.647 INFO:tasks.workunit.client.1.smithi169.stdout:0/36: chown d5/fa 180364 1 2024-09-10T03:42:29.647 INFO:tasks.workunit.client.1.smithi169.stdout:0/37: readlink l2 0 2024-09-10T03:42:29.647 INFO:tasks.workunit.client.1.smithi169.stdout:8/19: creat d3/f4 x:0 0 0 2024-09-10T03:42:29.648 INFO:tasks.workunit.client.1.smithi169.stdout:0/38: symlink d5/db/lc 0 2024-09-10T03:42:29.648 INFO:tasks.workunit.client.1.smithi169.stdout:8/20: mknod d3/c5 0 2024-09-10T03:42:29.650 INFO:tasks.workunit.client.1.smithi169.stdout:0/39: creat d5/fd x:0 0 0 2024-09-10T03:42:29.651 INFO:tasks.workunit.client.1.smithi169.stdout:8/21: unlink d3/c5 0 2024-09-10T03:42:29.654 INFO:tasks.workunit.client.1.smithi169.stdout:0/40: creat d5/fe x:0 0 0 2024-09-10T03:42:29.662 INFO:tasks.workunit.client.1.smithi169.stdout:4/23: dwrite d0/f1 [0,4194304] 0 2024-09-10T03:42:29.664 INFO:tasks.workunit.client.1.smithi169.stdout:4/24: creat d0/d2/f8 x:0 0 0 2024-09-10T03:42:29.666 INFO:tasks.workunit.client.1.smithi169.stdout:4/25: creat d0/d2/f9 x:0 0 0 2024-09-10T03:42:29.668 INFO:tasks.workunit.client.1.smithi169.stdout:4/26: creat d0/d6/fa x:0 0 0 2024-09-10T03:42:29.668 INFO:tasks.workunit.client.1.smithi169.stdout:4/27: write d0/d2/f5 [787092,95576] 0 2024-09-10T03:42:29.669 INFO:tasks.workunit.client.1.smithi169.stdout:4/28: rmdir d0/d2 39 2024-09-10T03:42:29.669 INFO:tasks.workunit.client.1.smithi169.stdout:3/27: write d0/f4 [4555805,65024] 0 2024-09-10T03:42:29.674 INFO:tasks.workunit.client.1.smithi169.stdout:0/41: dwrite d5/f9 [0,4194304] 0 2024-09-10T03:42:29.687 INFO:tasks.workunit.client.1.smithi169.stdout:0/42: dread d5/f9 [0,4194304] 0 2024-09-10T03:42:29.693 INFO:tasks.workunit.client.1.smithi169.stdout:3/28: dwrite d0/f4 [0,4194304] 0 2024-09-10T03:42:29.693 INFO:tasks.workunit.client.1.smithi169.stdout:3/29: fdatasync d0/f7 0 2024-09-10T03:42:29.693 INFO:tasks.workunit.client.1.smithi169.stdout:3/30: chown d0/f2 65438 1 2024-09-10T03:42:29.693 INFO:tasks.workunit.client.1.smithi169.stdout:3/31: chown d0/f2 1568 1 2024-09-10T03:42:29.704 INFO:tasks.workunit.client.1.smithi169.stdout:3/32: read d0/f2 [3020933,55339] 0 2024-09-10T03:42:29.705 INFO:tasks.workunit.client.1.smithi169.stdout:3/33: mkdir d0/dc 0 2024-09-10T03:42:29.706 INFO:tasks.workunit.client.1.smithi169.stdout:3/34: creat d0/fd x:0 0 0 2024-09-10T03:42:29.706 INFO:tasks.workunit.client.1.smithi169.stdout:3/35: stat d0/l5 0 2024-09-10T03:42:29.706 INFO:tasks.workunit.client.1.smithi169.stdout:3/36: write d0/f2 [3240585,18442] 0 2024-09-10T03:42:29.716 INFO:tasks.workunit.client.1.smithi169.stdout:3/37: dread d0/f4 [0,4194304] 0 2024-09-10T03:42:29.716 INFO:tasks.workunit.client.1.smithi169.stdout:3/38: write d0/f2 [5574345,110555] 0 2024-09-10T03:42:29.716 INFO:tasks.workunit.client.1.smithi169.stdout:3/39: creat d0/dc/fe x:0 0 0 2024-09-10T03:42:29.717 INFO:tasks.workunit.client.1.smithi169.stdout:3/40: mknod d0/dc/cf 0 2024-09-10T03:42:29.717 INFO:tasks.workunit.client.1.smithi169.stdout:3/41: chown d0/f7 7690659 1 2024-09-10T03:42:29.777 INFO:tasks.workunit.client.1.smithi169.stdout:9/19: getdents . 0 2024-09-10T03:42:29.780 INFO:tasks.workunit.client.1.smithi169.stdout:9/20: symlink l5 0 2024-09-10T03:42:29.780 INFO:tasks.workunit.client.1.smithi169.stdout:9/21: stat l5 0 2024-09-10T03:42:29.903 INFO:tasks.workunit.client.1.smithi169.stdout:2/25: sync 2024-09-10T03:42:29.904 INFO:tasks.workunit.client.1.smithi169.stdout:5/20: dread f2 [0,4194304] 0 2024-09-10T03:42:29.907 INFO:tasks.workunit.client.1.smithi169.stdout:5/21: dread f2 [0,4194304] 0 2024-09-10T03:42:29.907 INFO:tasks.workunit.client.1.smithi169.stdout:5/22: fdatasync f2 0 2024-09-10T03:42:29.909 INFO:tasks.workunit.client.1.smithi169.stdout:2/26: rename d0/d5/f6 to d0/d5/fa 0 2024-09-10T03:42:29.911 INFO:tasks.workunit.client.1.smithi169.stdout:2/27: fdatasync d0/d5/f7 0 2024-09-10T03:42:29.915 INFO:tasks.workunit.client.1.smithi169.stdout:2/28: creat d0/fb x:0 0 0 2024-09-10T03:42:29.915 INFO:tasks.workunit.client.1.smithi169.stdout:2/29: readlink - no filename 2024-09-10T03:42:29.916 INFO:tasks.workunit.client.1.smithi169.stdout:8/22: truncate f2 276182 0 2024-09-10T03:42:29.918 INFO:tasks.workunit.client.1.smithi169.stdout:2/30: creat d0/fc x:0 0 0 2024-09-10T03:42:29.920 INFO:tasks.workunit.client.1.smithi169.stdout:8/23: symlink d3/l6 0 2024-09-10T03:42:29.923 INFO:tasks.workunit.client.1.smithi169.stdout:2/31: creat d0/fd x:0 0 0 2024-09-10T03:42:29.923 INFO:tasks.workunit.client.1.smithi169.stdout:2/32: write d0/fb [2203,51186] 0 2024-09-10T03:42:29.923 INFO:tasks.workunit.client.1.smithi169.stdout:2/33: write d0/f2 [652280,40485] 0 2024-09-10T03:42:29.923 INFO:tasks.workunit.client.1.smithi169.stdout:2/34: write d0/fd [638905,73275] 0 2024-09-10T03:42:29.924 INFO:tasks.workunit.client.1.smithi169.stdout:5/23: dwrite d3/f5 [0,4194304] 0 2024-09-10T03:42:29.924 INFO:tasks.workunit.client.1.smithi169.stdout:8/24: creat d3/f7 x:0 0 0 2024-09-10T03:42:29.924 INFO:tasks.workunit.client.1.smithi169.stdout:8/25: truncate d3/f4 1026886 0 2024-09-10T03:42:29.931 INFO:tasks.workunit.client.1.smithi169.stdout:5/24: creat d3/f6 x:0 0 0 2024-09-10T03:42:29.931 INFO:tasks.workunit.client.1.smithi169.stdout:5/25: chown d3/f5 1810596133 1 2024-09-10T03:42:29.934 INFO:tasks.workunit.client.1.smithi169.stdout:8/26: mkdir d3/d8 0 2024-09-10T03:42:29.935 INFO:tasks.workunit.client.1.smithi169.stdout:8/27: chown d3 16 1 2024-09-10T03:42:29.935 INFO:tasks.workunit.client.1.smithi169.stdout:3/42: truncate d0/f2 4759354 0 2024-09-10T03:42:29.935 INFO:tasks.workunit.client.1.smithi169.stdout:3/43: write d0/f7 [657003,75899] 0 2024-09-10T03:42:29.937 INFO:tasks.workunit.client.1.smithi169.stdout:2/35: link d0/fd d0/d5/fe 0 2024-09-10T03:42:29.941 INFO:tasks.workunit.client.1.smithi169.stdout:5/26: symlink d3/l7 0 2024-09-10T03:42:29.943 INFO:tasks.workunit.client.1.smithi169.stdout:8/28: mknod d3/d8/c9 0 2024-09-10T03:42:29.947 INFO:tasks.workunit.client.1.smithi169.stdout:9/22: getdents . 0 2024-09-10T03:42:29.948 INFO:tasks.workunit.client.1.smithi169.stdout:2/36: creat d0/d5/ff x:0 0 0 2024-09-10T03:42:29.949 INFO:tasks.workunit.client.1.smithi169.stdout:2/37: write d0/d5/fe [704815,30610] 0 2024-09-10T03:42:29.951 INFO:tasks.workunit.client.1.smithi169.stdout:5/27: unlink d3/f5 0 2024-09-10T03:42:29.952 INFO:tasks.workunit.client.1.smithi169.stdout:8/29: mknod d3/ca 0 2024-09-10T03:42:29.955 INFO:tasks.workunit.client.1.smithi169.stdout:5/28: mkdir d3/d8 0 2024-09-10T03:42:29.955 INFO:tasks.workunit.client.1.smithi169.stdout:5/29: rename d3 to d3/d9 22 2024-09-10T03:42:29.956 INFO:tasks.workunit.client.1.smithi169.stdout:3/44: dwrite d0/f4 [0,4194304] 0 2024-09-10T03:42:29.956 INFO:tasks.workunit.client.1.smithi169.stdout:3/45: dread - d0/dc/fe zero size 2024-09-10T03:42:29.958 INFO:tasks.workunit.client.1.smithi169.stdout:2/38: dread d0/fd [0,4194304] 0 2024-09-10T03:42:29.959 INFO:tasks.workunit.client.1.smithi169.stdout:5/30: read f2 [1580193,35661] 0 2024-09-10T03:42:29.960 INFO:tasks.workunit.client.1.smithi169.stdout:8/30: creat d3/d8/fb x:0 0 0 2024-09-10T03:42:29.960 INFO:tasks.workunit.client.1.smithi169.stdout:8/31: chown d3/ca 12132683 1 2024-09-10T03:42:29.960 INFO:tasks.workunit.client.1.smithi169.stdout:9/23: mknod c6 0 2024-09-10T03:42:29.962 INFO:tasks.workunit.client.1.smithi169.stdout:5/31: dread f2 [0,4194304] 0 2024-09-10T03:42:29.965 INFO:tasks.workunit.client.1.smithi169.stdout:8/32: mkdir d3/dc 0 2024-09-10T03:42:29.968 INFO:tasks.workunit.client.1.smithi169.stdout:9/24: creat f7 x:0 0 0 2024-09-10T03:42:29.971 INFO:tasks.workunit.client.1.smithi169.stdout:5/32: rename d3/d8 to d3/da 0 2024-09-10T03:42:29.971 INFO:tasks.workunit.client.1.smithi169.stdout:5/33: write f2 [1916285,56593] 0 2024-09-10T03:42:29.972 INFO:tasks.workunit.client.1.smithi169.stdout:8/33: creat d3/fd x:0 0 0 2024-09-10T03:42:29.977 INFO:tasks.workunit.client.1.smithi169.stdout:5/34: dread f2 [0,4194304] 0 2024-09-10T03:42:29.977 INFO:tasks.workunit.client.1.smithi169.stdout:5/35: read f2 [1113148,46086] 0 2024-09-10T03:42:29.981 INFO:tasks.workunit.client.1.smithi169.stdout:5/36: rename f2 to d3/da/fb 0 2024-09-10T03:42:29.982 INFO:tasks.workunit.client.1.smithi169.stdout:5/37: dread - d3/f6 zero size 2024-09-10T03:42:30.119 INFO:tasks.workunit.client.1.smithi169.stdout:5/38: fsync d3/f6 0 2024-09-10T03:42:30.126 INFO:tasks.workunit.client.1.smithi169.stdout:4/29: sync 2024-09-10T03:42:30.126 INFO:tasks.workunit.client.1.smithi169.stdout:6/16: sync 2024-09-10T03:42:30.126 INFO:tasks.workunit.client.1.smithi169.stdout:1/14: sync 2024-09-10T03:42:30.126 INFO:tasks.workunit.client.1.smithi169.stdout:0/43: sync 2024-09-10T03:42:30.126 INFO:tasks.workunit.client.1.smithi169.stdout:7/16: sync 2024-09-10T03:42:30.126 INFO:tasks.workunit.client.1.smithi169.stdout:1/15: fsync f1 0 2024-09-10T03:42:30.129 INFO:tasks.workunit.client.1.smithi169.stdout:1/16: dread f1 [0,4194304] 0 2024-09-10T03:42:30.129 INFO:tasks.workunit.client.1.smithi169.stdout:4/30: creat d0/d6/d7/fb x:0 0 0 2024-09-10T03:42:30.131 INFO:tasks.workunit.client.1.smithi169.stdout:7/17: link d0/f2 d0/f3 0 2024-09-10T03:42:30.131 INFO:tasks.workunit.client.1.smithi169.stdout:7/18: write d0/f3 [1319030,90283] 0 2024-09-10T03:42:30.136 INFO:tasks.workunit.client.1.smithi169.stdout:6/17: creat d0/f8 x:0 0 0 2024-09-10T03:42:30.138 INFO:tasks.workunit.client.1.smithi169.stdout:4/31: creat d0/d6/d7/fc x:0 0 0 2024-09-10T03:42:30.140 INFO:tasks.workunit.client.1.smithi169.stdout:2/39: getdents d0/d5 0 2024-09-10T03:42:30.140 INFO:tasks.workunit.client.1.smithi169.stdout:2/40: chown d0/d5 179358 1 2024-09-10T03:42:30.140 INFO:tasks.workunit.client.1.smithi169.stdout:2/41: fsync d0/fc 0 2024-09-10T03:42:30.141 INFO:tasks.workunit.client.1.smithi169.stdout:6/18: rename d0/f3 to d0/f9 0 2024-09-10T03:42:30.144 INFO:tasks.workunit.client.1.smithi169.stdout:2/42: symlink d0/d5/l10 0 2024-09-10T03:42:30.146 INFO:tasks.workunit.client.1.smithi169.stdout:6/19: rename d0/d6/l7 to d0/d6/la 0 2024-09-10T03:42:30.148 INFO:tasks.workunit.client.1.smithi169.stdout:2/43: rename d0/d5/f7 to d0/f11 0 2024-09-10T03:42:30.148 INFO:tasks.workunit.client.1.smithi169.stdout:2/44: dread - d0/d5/ff zero size 2024-09-10T03:42:30.149 INFO:tasks.workunit.client.1.smithi169.stdout:6/20: creat d0/d6/fb x:0 0 0 2024-09-10T03:42:30.151 INFO:tasks.workunit.client.1.smithi169.stdout:4/32: read d0/f1 [4032463,46566] 0 2024-09-10T03:42:30.151 INFO:tasks.workunit.client.1.smithi169.stdout:4/33: write d0/d2/f9 [829552,80397] 0 2024-09-10T03:42:30.151 INFO:tasks.workunit.client.1.smithi169.stdout:6/21: creat d0/fc x:0 0 0 2024-09-10T03:42:30.151 INFO:tasks.workunit.client.1.smithi169.stdout:5/39: dwrite d3/f6 [0,4194304] 0 2024-09-10T03:42:30.152 INFO:tasks.workunit.client.1.smithi169.stdout:0/44: dwrite d5/f6 [0,4194304] 0 2024-09-10T03:42:30.152 INFO:tasks.workunit.client.1.smithi169.stdout:0/45: fsync d5/fa 0 2024-09-10T03:42:30.155 INFO:tasks.workunit.client.1.smithi169.stdout:6/22: mknod d0/cd 0 2024-09-10T03:42:30.156 INFO:tasks.workunit.client.1.smithi169.stdout:5/40: dread d3/da/fb [0,4194304] 0 2024-09-10T03:42:30.156 INFO:tasks.workunit.client.1.smithi169.stdout:5/41: chown d3/f6 8967 1 2024-09-10T03:42:30.156 INFO:tasks.workunit.client.1.smithi169.stdout:1/17: dwrite f1 [0,4194304] 0 2024-09-10T03:42:30.156 INFO:tasks.workunit.client.1.smithi169.stdout:1/18: fsync f1 0 2024-09-10T03:42:30.156 INFO:tasks.workunit.client.1.smithi169.stdout:1/19: rmdir - no directory 2024-09-10T03:42:30.158 INFO:tasks.workunit.client.1.smithi169.stdout:7/19: dread d0/f3 [0,4194304] 0 2024-09-10T03:42:30.158 INFO:tasks.workunit.client.1.smithi169.stdout:8/34: rename d3/d8 to d3/de 0 2024-09-10T03:42:30.158 INFO:tasks.workunit.client.1.smithi169.stdout:8/35: dread - d3/f7 zero size 2024-09-10T03:42:30.158 INFO:tasks.workunit.client.1.smithi169.stdout:8/36: truncate d3/de/fb 185225 0 2024-09-10T03:42:30.159 INFO:tasks.workunit.client.1.smithi169.stdout:8/37: dread d3/de/fb [0,4194304] 0 2024-09-10T03:42:30.166 INFO:tasks.workunit.client.1.smithi169.stdout:0/46: mkdir d5/df 0 2024-09-10T03:42:30.166 INFO:tasks.workunit.client.1.smithi169.stdout:0/47: truncate d5/fd 509816 0 2024-09-10T03:42:30.171 INFO:tasks.workunit.client.1.smithi169.stdout:6/23: creat d0/fe x:0 0 0 2024-09-10T03:42:30.171 INFO:tasks.workunit.client.1.smithi169.stdout:6/24: dread - d0/f8 zero size 2024-09-10T03:42:30.173 INFO:tasks.workunit.client.1.smithi169.stdout:9/25: getdents . 0 2024-09-10T03:42:30.173 INFO:tasks.workunit.client.1.smithi169.stdout:9/26: write f7 [735287,87992] 0 2024-09-10T03:42:30.177 INFO:tasks.workunit.client.1.smithi169.stdout:7/20: symlink d0/l4 0 2024-09-10T03:42:30.177 INFO:tasks.workunit.client.1.smithi169.stdout:7/21: chown d0/f3 59 1 2024-09-10T03:42:30.179 INFO:tasks.workunit.client.1.smithi169.stdout:7/22: dread d0/f3 [0,4194304] 0 2024-09-10T03:42:30.180 INFO:tasks.workunit.client.1.smithi169.stdout:0/48: rename d5/fa to d5/db/f10 0 2024-09-10T03:42:30.182 INFO:tasks.workunit.client.1.smithi169.stdout:6/25: mknod d0/cf 0 2024-09-10T03:42:30.186 INFO:tasks.workunit.client.1.smithi169.stdout:7/23: creat d0/f5 x:0 0 0 2024-09-10T03:42:30.190 INFO:tasks.workunit.client.1.smithi169.stdout:9/27: mknod c8 0 2024-09-10T03:42:30.190 INFO:tasks.workunit.client.1.smithi169.stdout:8/38: dwrite d3/fd [0,4194304] 0 2024-09-10T03:42:30.193 INFO:tasks.workunit.client.1.smithi169.stdout:7/24: creat d0/f6 x:0 0 0 2024-09-10T03:42:30.201 INFO:tasks.workunit.client.1.smithi169.stdout:7/25: symlink d0/l7 0 2024-09-10T03:42:30.201 INFO:tasks.workunit.client.1.smithi169.stdout:5/42: dwrite d3/da/fb [0,4194304] 0 2024-09-10T03:42:30.201 INFO:tasks.workunit.client.1.smithi169.stdout:4/34: dwrite d0/d6/d7/fc [0,4194304] 0 2024-09-10T03:42:30.202 INFO:tasks.workunit.client.1.smithi169.stdout:1/20: dwrite f1 [0,4194304] 0 2024-09-10T03:42:30.202 INFO:tasks.workunit.client.1.smithi169.stdout:0/49: dwrite d5/f9 [4194304,4194304] 0 2024-09-10T03:42:30.202 INFO:tasks.workunit.client.1.smithi169.stdout:0/50: stat d5 0 2024-09-10T03:42:30.202 INFO:tasks.workunit.client.1.smithi169.stdout:0/51: chown d5/fe 3791 1 2024-09-10T03:42:30.203 INFO:tasks.workunit.client.1.smithi169.stdout:7/26: rename d0/f6 to d0/f8 0 2024-09-10T03:42:30.204 INFO:tasks.workunit.client.1.smithi169.stdout:9/28: dread f7 [0,4194304] 0 2024-09-10T03:42:30.206 INFO:tasks.workunit.client.1.smithi169.stdout:7/27: dread d0/f2 [0,4194304] 0 2024-09-10T03:42:30.210 INFO:tasks.workunit.client.1.smithi169.stdout:5/43: creat d3/fc x:0 0 0 2024-09-10T03:42:30.213 INFO:tasks.workunit.client.1.smithi169.stdout:4/35: rename d0/f1 to d0/d6/d7/fd 0 2024-09-10T03:42:30.216 INFO:tasks.workunit.client.1.smithi169.stdout:0/52: mknod d5/df/c11 0 2024-09-10T03:42:30.218 INFO:tasks.workunit.client.1.smithi169.stdout:5/44: creat d3/da/fd x:0 0 0 2024-09-10T03:42:30.241 INFO:tasks.workunit.client.1.smithi169.stdout:5/45: dread d3/da/fb [0,4194304] 0 2024-09-10T03:42:30.241 INFO:tasks.workunit.client.1.smithi169.stdout:1/21: dwrite f1 [0,4194304] 0 2024-09-10T03:42:30.243 INFO:tasks.workunit.client.1.smithi169.stdout:5/46: mkdir d3/de 0 2024-09-10T03:42:30.245 INFO:tasks.workunit.client.1.smithi169.stdout:5/47: mkdir d3/da/df 0 2024-09-10T03:42:30.245 INFO:tasks.workunit.client.1.smithi169.stdout:5/48: read - d3/fc zero size 2024-09-10T03:42:30.245 INFO:tasks.workunit.client.1.smithi169.stdout:5/49: write d3/da/fd [529365,69479] 0 2024-09-10T03:42:30.245 INFO:tasks.workunit.client.1.smithi169.stdout:5/50: write d3/da/fd [318313,37351] 0 2024-09-10T03:42:30.248 INFO:tasks.workunit.client.1.smithi169.stdout:9/29: dwrite f7 [0,4194304] 0 2024-09-10T03:42:30.248 INFO:tasks.workunit.client.1.smithi169.stdout:9/30: chown c6 1 1 2024-09-10T03:42:30.248 INFO:tasks.workunit.client.1.smithi169.stdout:9/31: chown c3 14 1 2024-09-10T03:42:30.248 INFO:tasks.workunit.client.1.smithi169.stdout:7/28: dwrite d0/f8 [0,4194304] 0 2024-09-10T03:42:30.249 INFO:tasks.workunit.client.1.smithi169.stdout:4/36: dwrite d0/d6/d7/fb [0,4194304] 0 2024-09-10T03:42:30.249 INFO:tasks.workunit.client.1.smithi169.stdout:4/37: chown d0/d6/d7/fd 981 1 2024-09-10T03:42:30.249 INFO:tasks.workunit.client.1.smithi169.stdout:4/38: chown d0/c4 0 1 2024-09-10T03:42:30.249 INFO:tasks.workunit.client.1.smithi169.stdout:4/39: write d0/d6/d7/fd [1675360,112364] 0 2024-09-10T03:42:30.250 INFO:tasks.workunit.client.1.smithi169.stdout:0/53: dwrite f4 [0,4194304] 0 2024-09-10T03:42:30.254 INFO:tasks.workunit.client.1.smithi169.stdout:1/22: write f1 [695608,8038] 0 2024-09-10T03:42:30.254 INFO:tasks.workunit.client.1.smithi169.stdout:1/23: stat f1 0 2024-09-10T03:42:30.263 INFO:tasks.workunit.client.1.smithi169.stdout:4/40: creat d0/d2/fe x:0 0 0 2024-09-10T03:42:30.264 INFO:tasks.workunit.client.1.smithi169.stdout:0/54: mknod d5/c12 0 2024-09-10T03:42:30.267 INFO:tasks.workunit.client.1.smithi169.stdout:9/32: symlink l9 0 2024-09-10T03:42:30.269 INFO:tasks.workunit.client.1.smithi169.stdout:4/41: link d0/c4 d0/d2/cf 0 2024-09-10T03:42:30.270 INFO:tasks.workunit.client.1.smithi169.stdout:0/55: rmdir d5/db 39 2024-09-10T03:42:30.274 INFO:tasks.workunit.client.1.smithi169.stdout:0/56: mkdir d5/df/d13 0 2024-09-10T03:42:30.275 INFO:tasks.workunit.client.1.smithi169.stdout:7/29: dwrite d0/f8 [0,4194304] 0 2024-09-10T03:42:30.276 INFO:tasks.workunit.client.1.smithi169.stdout:9/33: symlink la 0 2024-09-10T03:42:30.279 INFO:tasks.workunit.client.1.smithi169.stdout:7/30: mknod d0/c9 0 2024-09-10T03:42:30.283 INFO:tasks.workunit.client.1.smithi169.stdout:1/24: dwrite f1 [0,4194304] 0 2024-09-10T03:42:30.285 INFO:tasks.workunit.client.1.smithi169.stdout:1/25: creat f2 x:0 0 0 2024-09-10T03:42:30.287 INFO:tasks.workunit.client.1.smithi169.stdout:1/26: rename f2 to f3 0 2024-09-10T03:42:30.288 INFO:tasks.workunit.client.1.smithi169.stdout:0/57: dread f4 [0,4194304] 0 2024-09-10T03:42:30.288 INFO:tasks.workunit.client.1.smithi169.stdout:0/58: readlink d5/db/lc 0 2024-09-10T03:42:30.289 INFO:tasks.workunit.client.1.smithi169.stdout:1/27: creat f4 x:0 0 0 2024-09-10T03:42:30.290 INFO:tasks.workunit.client.1.smithi169.stdout:0/59: creat d5/df/f14 x:0 0 0 2024-09-10T03:42:30.292 INFO:tasks.workunit.client.1.smithi169.stdout:1/28: rename f4 to f5 0 2024-09-10T03:42:30.294 INFO:tasks.workunit.client.1.smithi169.stdout:1/29: symlink l6 0 2024-09-10T03:42:30.295 INFO:tasks.workunit.client.1.smithi169.stdout:0/60: read f4 [3477950,115416] 0 2024-09-10T03:42:30.303 INFO:tasks.workunit.client.1.smithi169.stdout:1/30: dread f1 [0,4194304] 0 2024-09-10T03:42:30.303 INFO:tasks.workunit.client.1.smithi169.stdout:1/31: truncate f5 471427 0 2024-09-10T03:42:30.305 INFO:tasks.workunit.client.1.smithi169.stdout:1/32: creat f7 x:0 0 0 2024-09-10T03:42:30.310 INFO:tasks.workunit.client.1.smithi169.stdout:0/61: dwrite d5/f9 [4194304,4194304] 0 2024-09-10T03:42:30.311 INFO:tasks.workunit.client.1.smithi169.stdout:0/62: symlink d5/df/d13/l15 0 2024-09-10T03:42:30.316 INFO:tasks.workunit.client.1.smithi169.stdout:0/63: rename d5/fd to d5/df/d13/f16 0 2024-09-10T03:42:30.316 INFO:tasks.workunit.client.1.smithi169.stdout:0/64: write d5/fe [808454,71027] 0 2024-09-10T03:42:30.316 INFO:tasks.workunit.client.1.smithi169.stdout:0/65: rename d5/db to d5/db/d17 22 2024-09-10T03:42:30.319 INFO:tasks.workunit.client.1.smithi169.stdout:0/66: symlink d5/l18 0 2024-09-10T03:42:30.321 INFO:tasks.workunit.client.1.smithi169.stdout:0/67: creat d5/df/d13/f19 x:0 0 0 2024-09-10T03:42:30.321 INFO:tasks.workunit.client.1.smithi169.stdout:0/68: readlink d5/l18 0 2024-09-10T03:42:30.322 INFO:tasks.workunit.client.1.smithi169.stdout:0/69: stat c3 0 2024-09-10T03:42:30.322 INFO:tasks.workunit.client.1.smithi169.stdout:1/33: dwrite f3 [0,4194304] 0 2024-09-10T03:42:30.327 INFO:tasks.workunit.client.1.smithi169.stdout:1/34: unlink f1 0 2024-09-10T03:42:30.331 INFO:tasks.workunit.client.1.smithi169.stdout:1/35: creat f8 x:0 0 0 2024-09-10T03:42:30.344 INFO:tasks.workunit.client.1.smithi169.stdout:0/70: dwrite d5/df/d13/f19 [0,4194304] 0 2024-09-10T03:42:30.348 INFO:tasks.workunit.client.1.smithi169.stdout:0/71: symlink d5/df/l1a 0 2024-09-10T03:42:30.348 INFO:tasks.workunit.client.1.smithi169.stdout:0/72: dread - d5/df/f14 zero size 2024-09-10T03:42:30.360 INFO:tasks.workunit.client.1.smithi169.stdout:6/26: unlink d0/d6/la 0 2024-09-10T03:42:30.360 INFO:tasks.workunit.client.1.smithi169.stdout:6/27: truncate d0/fe 257046 0 2024-09-10T03:42:30.362 INFO:tasks.workunit.client.1.smithi169.stdout:0/73: dwrite d5/df/d13/f16 [0,4194304] 0 2024-09-10T03:42:30.363 INFO:tasks.workunit.client.1.smithi169.stdout:2/45: getdents d0 0 2024-09-10T03:42:30.363 INFO:tasks.workunit.client.1.smithi169.stdout:2/46: write d0/d5/fe [1445597,93030] 0 2024-09-10T03:42:30.364 INFO:tasks.workunit.client.1.smithi169.stdout:2/47: write d0/f2 [1570223,27662] 0 2024-09-10T03:42:30.371 INFO:tasks.workunit.client.1.smithi169.stdout:8/39: getdents d3 0 2024-09-10T03:42:30.371 INFO:tasks.workunit.client.1.smithi169.stdout:3/46: sync 2024-09-10T03:42:30.373 INFO:tasks.workunit.client.1.smithi169.stdout:8/40: rename d3/fd to d3/de/ff 0 2024-09-10T03:42:30.375 INFO:tasks.workunit.client.1.smithi169.stdout:3/47: unlink d0/fd 0 2024-09-10T03:42:30.375 INFO:tasks.workunit.client.1.smithi169.stdout:3/48: truncate d0/dc/fe 499080 0 2024-09-10T03:42:30.375 INFO:tasks.workunit.client.1.smithi169.stdout:3/49: chown d0/dc/cf 4299059 1 2024-09-10T03:42:30.375 INFO:tasks.workunit.client.1.smithi169.stdout:8/41: dread f2 [0,4194304] 0 2024-09-10T03:42:30.378 INFO:tasks.workunit.client.1.smithi169.stdout:3/50: creat d0/f10 x:0 0 0 2024-09-10T03:42:30.378 INFO:tasks.workunit.client.1.smithi169.stdout:3/51: chown d0/f7 3482788 1 2024-09-10T03:42:30.379 INFO:tasks.workunit.client.1.smithi169.stdout:8/42: mkdir d3/dc/d10 0 2024-09-10T03:42:30.379 INFO:tasks.workunit.client.1.smithi169.stdout:8/43: fdatasync d3/f7 0 2024-09-10T03:42:30.379 INFO:tasks.workunit.client.1.smithi169.stdout:8/44: stat d3/de/ff 0 2024-09-10T03:42:30.379 INFO:tasks.workunit.client.1.smithi169.stdout:8/45: stat d3/de 0 2024-09-10T03:42:30.380 INFO:tasks.workunit.client.1.smithi169.stdout:8/46: truncate d3/de/fb 1005808 0 2024-09-10T03:42:30.389 INFO:tasks.workunit.client.1.smithi169.stdout:2/48: dwrite d0/fd [0,4194304] 0 2024-09-10T03:42:30.391 INFO:tasks.workunit.client.1.smithi169.stdout:6/28: dwrite d0/f9 [0,4194304] 0 2024-09-10T03:42:30.391 INFO:tasks.workunit.client.1.smithi169.stdout:6/29: fsync d0/fe 0 2024-09-10T03:42:30.393 INFO:tasks.workunit.client.1.smithi169.stdout:2/49: symlink d0/l12 0 2024-09-10T03:42:30.396 INFO:tasks.workunit.client.1.smithi169.stdout:6/30: rmdir d0/d6 39 2024-09-10T03:42:30.399 INFO:tasks.workunit.client.1.smithi169.stdout:2/50: creat d0/d5/f13 x:0 0 0 2024-09-10T03:42:30.400 INFO:tasks.workunit.client.1.smithi169.stdout:2/51: write d0/f2 [1430707,67675] 0 2024-09-10T03:42:30.410 INFO:tasks.workunit.client.1.smithi169.stdout:7/31: rename d0/f8 to d0/fa 0 2024-09-10T03:42:30.410 INFO:tasks.workunit.client.1.smithi169.stdout:7/32: stat d0 0 2024-09-10T03:42:30.425 INFO:tasks.workunit.client.1.smithi169.stdout:2/52: dwrite d0/fb [0,4194304] 0 2024-09-10T03:42:30.425 INFO:tasks.workunit.client.1.smithi169.stdout:2/53: write d0/d5/fa [4085581,46413] 0 2024-09-10T03:42:30.440 INFO:tasks.workunit.client.1.smithi169.stdout:7/33: dwrite d0/f5 [0,4194304] 0 2024-09-10T03:42:30.442 INFO:tasks.workunit.client.1.smithi169.stdout:2/54: dwrite d0/fc [0,4194304] 0 2024-09-10T03:42:30.444 INFO:tasks.workunit.client.1.smithi169.stdout:7/34: mkdir d0/db 0 2024-09-10T03:42:30.444 INFO:tasks.workunit.client.1.smithi169.stdout:7/35: chown d0/l4 301272186 1 2024-09-10T03:42:30.448 INFO:tasks.workunit.client.1.smithi169.stdout:5/51: truncate d3/da/fd 427545 0 2024-09-10T03:42:30.450 INFO:tasks.workunit.client.1.smithi169.stdout:2/55: dread d0/fd [0,4194304] 0 2024-09-10T03:42:30.450 INFO:tasks.workunit.client.1.smithi169.stdout:2/56: stat d0/d5 0 2024-09-10T03:42:30.456 INFO:tasks.workunit.client.1.smithi169.stdout:4/42: truncate d0/d6/d7/fb 3979307 0 2024-09-10T03:42:30.456 INFO:tasks.workunit.client.1.smithi169.stdout:4/43: truncate d0/d6/d7/fd 4365686 0 2024-09-10T03:42:30.458 INFO:tasks.workunit.client.1.smithi169.stdout:4/44: unlink d0/d6/d7/fd 0 2024-09-10T03:42:30.458 INFO:tasks.workunit.client.1.smithi169.stdout:4/45: stat d0/d2/f5 0 2024-09-10T03:42:30.458 INFO:tasks.workunit.client.1.smithi169.stdout:4/46: chown d0 3229644 1 2024-09-10T03:42:30.467 INFO:tasks.workunit.client.1.smithi169.stdout:9/34: getdents . 0 2024-09-10T03:42:30.479 INFO:tasks.workunit.client.1.smithi169.stdout:2/57: dwrite d0/d5/fa [4194304,4194304] 0 2024-09-10T03:42:30.480 INFO:tasks.workunit.client.1.smithi169.stdout:9/35: write f7 [3355890,30764] 0 2024-09-10T03:42:30.482 INFO:tasks.workunit.client.1.smithi169.stdout:2/58: mknod d0/c14 0 2024-09-10T03:42:30.485 INFO:tasks.workunit.client.1.smithi169.stdout:2/59: dread d0/fd [0,4194304] 0 2024-09-10T03:42:30.486 INFO:tasks.workunit.client.1.smithi169.stdout:2/60: chown d0/d5/fa 6 1 2024-09-10T03:42:30.486 INFO:tasks.workunit.client.1.smithi169.stdout:9/36: mkdir db 0 2024-09-10T03:42:30.494 INFO:tasks.workunit.client.1.smithi169.stdout:1/36: fsync f7 0 2024-09-10T03:42:30.496 INFO:tasks.workunit.client.1.smithi169.stdout:1/37: mknod c9 0 2024-09-10T03:42:30.502 INFO:tasks.workunit.client.1.smithi169.stdout:1/38: write f3 [2290187,71069] 0 2024-09-10T03:42:30.502 INFO:tasks.workunit.client.1.smithi169.stdout:1/39: truncate f7 995790 0 2024-09-10T03:42:30.502 INFO:tasks.workunit.client.1.smithi169.stdout:1/40: read - f8 zero size 2024-09-10T03:42:30.517 INFO:tasks.workunit.client.1.smithi169.stdout:1/41: dwrite f5 [0,4194304] 0 2024-09-10T03:42:30.520 INFO:tasks.workunit.client.1.smithi169.stdout:1/42: creat fa x:0 0 0 2024-09-10T03:42:30.521 INFO:tasks.workunit.client.1.smithi169.stdout:1/43: fsync f5 0 2024-09-10T03:42:30.522 INFO:tasks.workunit.client.1.smithi169.stdout:1/44: creat fb x:0 0 0 2024-09-10T03:42:30.524 INFO:tasks.workunit.client.1.smithi169.stdout:1/45: symlink lc 0 2024-09-10T03:42:30.524 INFO:tasks.workunit.client.1.smithi169.stdout:1/46: readlink l6 0 2024-09-10T03:42:30.531 INFO:tasks.workunit.client.1.smithi169.stdout:1/47: write f5 [924997,109184] 0 2024-09-10T03:42:30.533 INFO:tasks.workunit.client.1.smithi169.stdout:1/48: symlink ld 0 2024-09-10T03:42:30.534 INFO:tasks.workunit.client.1.smithi169.stdout:1/49: dread - fa zero size 2024-09-10T03:42:30.540 INFO:tasks.workunit.client.1.smithi169.stdout:6/31: getdents d0/d6 0 2024-09-10T03:42:30.547 INFO:tasks.workunit.client.1.smithi169.stdout:1/50: link l6 le 0 2024-09-10T03:42:30.549 INFO:tasks.workunit.client.1.smithi169.stdout:6/32: mkdir d0/d10 0 2024-09-10T03:42:30.549 INFO:tasks.workunit.client.1.smithi169.stdout:0/74: truncate d5/f9 6740017 0 2024-09-10T03:42:30.551 INFO:tasks.workunit.client.1.smithi169.stdout:6/33: link d0/fe d0/d6/f11 0 2024-09-10T03:42:30.552 INFO:tasks.workunit.client.1.smithi169.stdout:6/34: dread d0/d6/f11 [0,4194304] 0 2024-09-10T03:42:30.552 INFO:tasks.workunit.client.1.smithi169.stdout:8/47: rename d3/de/ff to d3/dc/f11 0 2024-09-10T03:42:30.552 INFO:tasks.workunit.client.1.smithi169.stdout:7/36: rename d0 to d0/dc 22 2024-09-10T03:42:30.554 INFO:tasks.workunit.client.1.smithi169.stdout:3/52: getdents d0 0 2024-09-10T03:42:30.555 INFO:tasks.workunit.client.1.smithi169.stdout:0/75: unlink l1 0 2024-09-10T03:42:30.556 INFO:tasks.workunit.client.1.smithi169.stdout:0/76: chown d5 297671 1 2024-09-10T03:42:30.556 INFO:tasks.workunit.client.1.smithi169.stdout:6/35: mkdir d0/d12 0 2024-09-10T03:42:30.559 INFO:tasks.workunit.client.1.smithi169.stdout:8/48: rmdir d3/dc/d10 0 2024-09-10T03:42:30.560 INFO:tasks.workunit.client.1.smithi169.stdout:7/37: rmdir d0/db 0 2024-09-10T03:42:30.561 INFO:tasks.workunit.client.1.smithi169.stdout:7/38: read d0/f3 [1125621,67585] 0 2024-09-10T03:42:30.563 INFO:tasks.workunit.client.1.smithi169.stdout:3/53: symlink d0/l11 0 2024-09-10T03:42:30.564 INFO:tasks.workunit.client.1.smithi169.stdout:0/77: mkdir d5/df/d1b 0 2024-09-10T03:42:30.566 INFO:tasks.workunit.client.1.smithi169.stdout:8/49: rmdir d3/de 39 2024-09-10T03:42:30.567 INFO:tasks.workunit.client.1.smithi169.stdout:7/39: rmdir d0 39 2024-09-10T03:42:30.570 INFO:tasks.workunit.client.1.smithi169.stdout:0/78: rename d5/l18 to d5/df/d1b/l1c 0 2024-09-10T03:42:30.570 INFO:tasks.workunit.client.1.smithi169.stdout:0/79: fsync d5/df/d13/f16 0 2024-09-10T03:42:30.570 INFO:tasks.workunit.client.1.smithi169.stdout:0/80: readlink d5/df/d13/l15 0 2024-09-10T03:42:30.572 INFO:tasks.workunit.client.1.smithi169.stdout:3/54: creat d0/dc/f12 x:0 0 0 2024-09-10T03:42:30.572 INFO:tasks.workunit.client.1.smithi169.stdout:3/55: fdatasync d0/dc/fe 0 2024-09-10T03:42:30.574 INFO:tasks.workunit.client.1.smithi169.stdout:8/50: write d3/dc/f11 [1898355,117571] 0 2024-09-10T03:42:30.575 INFO:tasks.workunit.client.1.smithi169.stdout:0/81: creat d5/df/d13/f1d x:0 0 0 2024-09-10T03:42:30.576 INFO:tasks.workunit.client.1.smithi169.stdout:3/56: creat d0/f13 x:0 0 0 2024-09-10T03:42:30.579 INFO:tasks.workunit.client.1.smithi169.stdout:8/51: symlink d3/de/l12 0 2024-09-10T03:42:30.580 INFO:tasks.workunit.client.1.smithi169.stdout:0/82: creat d5/df/f1e x:0 0 0 2024-09-10T03:42:30.581 INFO:tasks.workunit.client.1.smithi169.stdout:6/36: dwrite d0/fc [0,4194304] 0 2024-09-10T03:42:30.581 INFO:tasks.workunit.client.1.smithi169.stdout:6/37: stat d0/cd 0 2024-09-10T03:42:30.582 INFO:tasks.workunit.client.1.smithi169.stdout:3/57: truncate d0/fa 1660466 0 2024-09-10T03:42:30.583 INFO:tasks.workunit.client.1.smithi169.stdout:0/83: mknod d5/df/d1b/c1f 0 2024-09-10T03:42:30.583 INFO:tasks.workunit.client.1.smithi169.stdout:0/84: truncate d5/df/d13/f19 4786057 0 2024-09-10T03:42:30.589 INFO:tasks.workunit.client.1.smithi169.stdout:6/38: rename d0/d6/f11 to d0/d6/f13 0 2024-09-10T03:42:30.591 INFO:tasks.workunit.client.1.smithi169.stdout:6/39: dread d0/fe [0,4194304] 0 2024-09-10T03:42:30.591 INFO:tasks.workunit.client.1.smithi169.stdout:6/40: stat d0 0 2024-09-10T03:42:30.591 INFO:tasks.workunit.client.1.smithi169.stdout:6/41: write d0/fe [341058,93180] 0 2024-09-10T03:42:30.591 INFO:tasks.workunit.client.1.smithi169.stdout:6/42: readlink d0/l2 0 2024-09-10T03:42:30.591 INFO:tasks.workunit.client.1.smithi169.stdout:6/43: dread - d0/d6/fb zero size 2024-09-10T03:42:30.591 INFO:tasks.workunit.client.1.smithi169.stdout:6/44: truncate d0/f8 126406 0 2024-09-10T03:42:30.593 INFO:tasks.workunit.client.1.smithi169.stdout:2/61: truncate d0/fb 534271 0 2024-09-10T03:42:30.593 INFO:tasks.workunit.client.1.smithi169.stdout:2/62: stat d0/fb 0 2024-09-10T03:42:30.594 INFO:tasks.workunit.client.1.smithi169.stdout:8/52: dwrite d3/de/fb [0,4194304] 0 2024-09-10T03:42:30.596 INFO:tasks.workunit.client.1.smithi169.stdout:3/58: rename d0/f2 to d0/dc/f14 0 2024-09-10T03:42:30.604 INFO:tasks.workunit.client.1.smithi169.stdout:3/59: link d0/dc/fe d0/dc/f15 0 2024-09-10T03:42:30.605 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:30 smithi169 ceph-mon[35668]: mgrmap e22: smithi169.bkwgjl(active, since 1.00918s) 2024-09-10T03:42:30.605 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:30 smithi169 ceph-mon[35668]: pgmap v3: 65 pgs: 65 active+clean; 206 MiB data, 894 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:42:30.605 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:30 smithi169 ceph-mon[35668]: from='mgr.24369 ' entity='mgr.smithi169.bkwgjl' 2024-09-10T03:42:30.605 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:30 smithi169 ceph-mon[35668]: from='mgr.24369 ' entity='mgr.smithi169.bkwgjl' 2024-09-10T03:42:30.606 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:30 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:42:30.606 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:30 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:42:30.606 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:30 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-10T03:42:30.606 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:30 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.bkwgjl/mirror_snapshot_schedule"}]: dispatch 2024-09-10T03:42:30.606 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:30 smithi169 ceph-mon[35668]: from='mgr.24369 ' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.bkwgjl/mirror_snapshot_schedule"}]: dispatch 2024-09-10T03:42:30.606 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:30 smithi169 ceph-mon[35668]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.bkwgjl/trash_purge_schedule"}]: dispatch 2024-09-10T03:42:30.606 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:30 smithi169 ceph-mon[35668]: from='mgr.24369 ' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.bkwgjl/trash_purge_schedule"}]: dispatch 2024-09-10T03:42:30.617 INFO:tasks.workunit.client.1.smithi169.stdout:6/45: dwrite d0/f9 [0,4194304] 0 2024-09-10T03:42:30.618 INFO:tasks.workunit.client.1.smithi169.stdout:0/85: dwrite d5/f6 [0,4194304] 0 2024-09-10T03:42:30.618 INFO:tasks.workunit.client.1.smithi169.stdout:3/60: dread d0/f4 [0,4194304] 0 2024-09-10T03:42:30.618 INFO:tasks.workunit.client.1.smithi169.stdout:3/61: write d0/dc/f15 [735837,71353] 0 2024-09-10T03:42:30.618 INFO:tasks.workunit.client.1.smithi169.stdout:3/62: stat d0/f4 0 2024-09-10T03:42:30.619 INFO:tasks.workunit.client.1.smithi169.stdout:6/46: write d0/d6/f13 [36976,45203] 0 2024-09-10T03:42:30.623 INFO:tasks.workunit.client.1.smithi169.stdout:0/86: truncate d5/df/d13/f16 3027635 0 2024-09-10T03:42:30.625 INFO:tasks.workunit.client.1.smithi169.stdout:3/63: dread d0/dc/f14 [0,4194304] 0 2024-09-10T03:42:30.625 INFO:tasks.workunit.client.1.smithi169.stdout:3/64: readlink d0/l11 0 2024-09-10T03:42:30.628 INFO:tasks.workunit.client.1.smithi169.stdout:6/47: write d0/f9 [3253015,51493] 0 2024-09-10T03:42:30.629 INFO:tasks.workunit.client.1.smithi169.stdout:5/52: dwrite d3/da/fd [0,4194304] 0 2024-09-10T03:42:30.633 INFO:tasks.workunit.client.1.smithi169.stdout:6/48: link d0/d6/fb d0/f14 0 2024-09-10T03:42:30.633 INFO:tasks.workunit.client.1.smithi169.stdout:6/49: fdatasync d0/d6/f13 0 2024-09-10T03:42:30.637 INFO:tasks.workunit.client.1.smithi169.stdout:5/53: mknod d3/da/df/c10 0 2024-09-10T03:42:30.642 INFO:tasks.workunit.client.1.smithi169.stdout:6/50: mkdir d0/d15 0 2024-09-10T03:42:30.645 INFO:tasks.workunit.client.1.smithi169.stdout:6/51: mknod d0/d10/c16 0 2024-09-10T03:42:30.650 INFO:tasks.workunit.client.1.smithi169.stdout:5/54: dwrite d3/fc [0,4194304] 0 2024-09-10T03:42:30.650 INFO:tasks.workunit.client.1.smithi169.stdout:9/37: truncate f7 57740 0 2024-09-10T03:42:30.656 INFO:tasks.workunit.client.1.smithi169.stdout:5/55: truncate d3/f6 2699884 0 2024-09-10T03:42:30.660 INFO:tasks.workunit.client.1.smithi169.stdout:9/38: creat db/fc x:0 0 0 2024-09-10T03:42:30.662 INFO:tasks.workunit.client.1.smithi169.stdout:9/39: symlink db/ld 0 2024-09-10T03:42:30.663 INFO:tasks.workunit.client.1.smithi169.stdout:5/56: write d3/da/fb [3648006,5771] 0 2024-09-10T03:42:30.665 INFO:tasks.workunit.client.1.smithi169.stdout:5/57: symlink d3/da/l11 0 2024-09-10T03:42:30.665 INFO:tasks.workunit.client.1.smithi169.stdout:5/58: chown d3/de 15 1 2024-09-10T03:42:30.671 INFO:tasks.workunit.client.1.smithi169.stdout:6/52: dwrite d0/f9 [4194304,4194304] 0 2024-09-10T03:42:30.671 INFO:tasks.workunit.client.1.smithi169.stdout:6/53: read d0/d6/f13 [45224,31798] 0 2024-09-10T03:42:30.675 INFO:tasks.workunit.client.1.smithi169.stdout:5/59: write d3/fc [1623694,54738] 0 2024-09-10T03:42:30.676 INFO:tasks.workunit.client.1.smithi169.stdout:6/54: creat d0/f17 x:0 0 0 2024-09-10T03:42:30.676 INFO:tasks.workunit.client.1.smithi169.stdout:6/55: truncate d0/d6/fb 872980 0 2024-09-10T03:42:30.683 INFO:tasks.workunit.client.1.smithi169.stdout:5/60: dread d3/da/fd [0,4194304] 0 2024-09-10T03:42:30.683 INFO:tasks.workunit.client.1.smithi169.stdout:5/61: chown d3/f6 45160 1 2024-09-10T03:42:30.700 INFO:tasks.workunit.client.1.smithi169.stdout:6/56: dwrite d0/f9 [0,4194304] 0 2024-09-10T03:42:30.751 INFO:tasks.workunit.client.1.smithi169.stdout:2/63: dread d0/fb [0,4194304] 0 2024-09-10T03:42:30.753 INFO:tasks.workunit.client.1.smithi169.stdout:2/64: getdents d0 0 2024-09-10T03:42:30.758 INFO:tasks.workunit.client.1.smithi169.stdout:3/65: fsync d0/dc/fe 0 2024-09-10T03:42:30.763 INFO:tasks.workunit.client.1.smithi169.stdout:3/66: dread d0/f4 [0,4194304] 0 2024-09-10T03:42:30.766 INFO:tasks.workunit.client.1.smithi169.stdout:3/67: dread d0/fa [0,4194304] 0 2024-09-10T03:42:30.769 INFO:tasks.workunit.client.1.smithi169.stdout:2/65: dwrite d0/d5/ff [0,4194304] 0 2024-09-10T03:42:30.769 INFO:tasks.workunit.client.1.smithi169.stdout:3/68: dread d0/f7 [0,4194304] 0 2024-09-10T03:42:30.770 INFO:tasks.workunit.client.1.smithi169.stdout:3/69: dread d0/dc/f14 [4194304,4194304] 0 2024-09-10T03:42:30.772 INFO:tasks.workunit.client.1.smithi169.stdout:2/66: creat d0/f15 x:0 0 0 2024-09-10T03:42:30.773 INFO:tasks.workunit.client.1.smithi169.stdout:3/70: creat d0/dc/f16 x:0 0 0 2024-09-10T03:42:30.776 INFO:tasks.workunit.client.1.smithi169.stdout:2/67: mkdir d0/d16 0 2024-09-10T03:42:30.778 INFO:tasks.workunit.client.1.smithi169.stdout:2/68: rmdir d0 39 2024-09-10T03:42:30.778 INFO:tasks.workunit.client.1.smithi169.stdout:2/69: chown d0/d5 1593122378 1 2024-09-10T03:42:30.779 INFO:tasks.workunit.client.1.smithi169.stdout:0/87: truncate d5/f6 3426072 0 2024-09-10T03:42:30.779 INFO:tasks.workunit.client.1.smithi169.stdout:3/71: dread d0/f4 [0,4194304] 0 2024-09-10T03:42:30.780 INFO:tasks.workunit.client.1.smithi169.stdout:3/72: stat d0/dc/f12 0 2024-09-10T03:42:30.780 INFO:tasks.workunit.client.1.smithi169.stdout:3/73: read - d0/dc/f16 zero size 2024-09-10T03:42:30.782 INFO:tasks.workunit.client.1.smithi169.stdout:0/88: mkdir d5/db/d20 0 2024-09-10T03:42:30.783 INFO:tasks.workunit.client.1.smithi169.stdout:3/74: getdents d0 0 2024-09-10T03:42:30.786 INFO:tasks.workunit.client.1.smithi169.stdout:0/89: rmdir d5/db 39 2024-09-10T03:42:30.791 INFO:tasks.workunit.client.1.smithi169.stdout:0/90: rename d5/df/f1e to d5/db/f21 0 2024-09-10T03:42:30.793 INFO:tasks.workunit.client.1.smithi169.stdout:2/70: dwrite d0/f2 [0,4194304] 0 2024-09-10T03:42:30.793 INFO:tasks.workunit.client.1.smithi169.stdout:2/71: chown d0/d5/fe 39299682 1 2024-09-10T03:42:30.793 INFO:tasks.workunit.client.1.smithi169.stdout:3/75: write d0/f10 [859382,70597] 0 2024-09-10T03:42:30.793 INFO:tasks.workunit.client.1.smithi169.stdout:3/76: chown d0 293116904 1 2024-09-10T03:42:30.794 INFO:tasks.workunit.client.1.smithi169.stdout:0/91: rmdir d5/db 39 2024-09-10T03:42:30.794 INFO:tasks.workunit.client.1.smithi169.stdout:0/92: stat d5/db/lc 0 2024-09-10T03:42:30.794 INFO:tasks.workunit.client.1.smithi169.stdout:0/93: chown d5/db/f10 0 1 2024-09-10T03:42:30.795 INFO:tasks.workunit.client.1.smithi169.stdout:3/77: read d0/dc/f15 [317326,21621] 0 2024-09-10T03:42:30.795 INFO:tasks.workunit.client.1.smithi169.stdout:3/78: write d0/fa [2678212,54883] 0 2024-09-10T03:42:30.800 INFO:tasks.workunit.client.1.smithi169.stdout:0/94: mknod d5/df/c22 0 2024-09-10T03:42:30.800 INFO:tasks.workunit.client.1.smithi169.stdout:3/79: mkdir d0/dc/d17 0 2024-09-10T03:42:30.802 INFO:tasks.workunit.client.1.smithi169.stdout:0/95: rmdir d5 39 2024-09-10T03:42:30.802 INFO:tasks.workunit.client.1.smithi169.stdout:0/96: readlink l2 0 2024-09-10T03:42:30.804 INFO:tasks.workunit.client.1.smithi169.stdout:3/80: rename d0/l11 to d0/dc/l18 0 2024-09-10T03:42:30.807 INFO:tasks.workunit.client.1.smithi169.stdout:0/97: creat d5/df/f23 x:0 0 0 2024-09-10T03:42:30.808 INFO:tasks.workunit.client.1.smithi169.stdout:4/47: sync 2024-09-10T03:42:30.808 INFO:tasks.workunit.client.1.smithi169.stdout:4/48: write d0/d2/f9 [1294368,126846] 0 2024-09-10T03:42:30.808 INFO:tasks.workunit.client.1.smithi169.stdout:5/62: fsync d3/f6 0 2024-09-10T03:42:30.809 INFO:tasks.workunit.client.1.smithi169.stdout:2/72: dwrite d0/fc [0,4194304] 0 2024-09-10T03:42:30.809 INFO:tasks.workunit.client.1.smithi169.stdout:2/73: stat d0/d5/fa 0 2024-09-10T03:42:30.811 INFO:tasks.workunit.client.1.smithi169.stdout:0/98: unlink d5/df/f23 0 2024-09-10T03:42:30.812 INFO:tasks.workunit.client.1.smithi169.stdout:4/49: creat d0/d6/d7/f10 x:0 0 0 2024-09-10T03:42:30.817 INFO:tasks.workunit.client.1.smithi169.stdout:2/74: mkdir d0/d17 0 2024-09-10T03:42:30.817 INFO:tasks.workunit.client.1.smithi169.stdout:9/40: dwrite f7 [0,4194304] 0 2024-09-10T03:42:30.819 INFO:tasks.workunit.client.1.smithi169.stdout:0/99: mkdir d5/db/d24 0 2024-09-10T03:42:30.822 INFO:tasks.workunit.client.1.smithi169.stdout:4/50: mkdir d0/d2/d11 0 2024-09-10T03:42:30.822 INFO:tasks.workunit.client.1.smithi169.stdout:4/51: dread - d0/d6/fa zero size 2024-09-10T03:42:30.822 INFO:tasks.workunit.client.1.smithi169.stdout:4/52: write d0/d2/f8 [70525,102382] 0 2024-09-10T03:42:30.822 INFO:tasks.workunit.client.1.smithi169.stdout:4/53: fdatasync d0/d6/d7/fc 0 2024-09-10T03:42:30.823 INFO:tasks.workunit.client.1.smithi169.stdout:2/75: rmdir d0 39 2024-09-10T03:42:30.823 INFO:tasks.workunit.client.1.smithi169.stdout:2/76: chown d0/d5/fe 115638 1 2024-09-10T03:42:30.823 INFO:tasks.workunit.client.1.smithi169.stdout:2/77: chown d0/d16 284212 1 2024-09-10T03:42:30.823 INFO:tasks.workunit.client.1.smithi169.stdout:6/57: fsync d0/f17 0 2024-09-10T03:42:30.825 INFO:tasks.workunit.client.1.smithi169.stdout:9/41: link db/fc db/fe 0 2024-09-10T03:42:30.829 INFO:tasks.workunit.client.1.smithi169.stdout:0/100: creat d5/db/f25 x:0 0 0 2024-09-10T03:42:30.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:30 smithi112 ceph-mon[27737]: mgrmap e22: smithi169.bkwgjl(active, since 1.00918s) 2024-09-10T03:42:30.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:30 smithi112 ceph-mon[27737]: pgmap v3: 65 pgs: 65 active+clean; 206 MiB data, 894 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:42:30.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:30 smithi112 ceph-mon[27737]: from='mgr.24369 ' entity='mgr.smithi169.bkwgjl' 2024-09-10T03:42:30.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:30 smithi112 ceph-mon[27737]: from='mgr.24369 ' entity='mgr.smithi169.bkwgjl' 2024-09-10T03:42:30.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:30 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T03:42:30.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:30 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T03:42:30.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:30 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-10T03:42:30.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:30 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.bkwgjl/mirror_snapshot_schedule"}]: dispatch 2024-09-10T03:42:30.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:30 smithi112 ceph-mon[27737]: from='mgr.24369 ' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.bkwgjl/mirror_snapshot_schedule"}]: dispatch 2024-09-10T03:42:30.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:30 smithi112 ceph-mon[27737]: from='mgr.24369 172.21.15.169:0/3058857466' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.bkwgjl/trash_purge_schedule"}]: dispatch 2024-09-10T03:42:30.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:30 smithi112 ceph-mon[27737]: from='mgr.24369 ' entity='mgr.smithi169.bkwgjl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi169.bkwgjl/trash_purge_schedule"}]: dispatch 2024-09-10T03:42:30.832 INFO:tasks.workunit.client.1.smithi169.stdout:4/54: link d0/d2/cf d0/d6/d7/c12 0 2024-09-10T03:42:30.834 INFO:tasks.workunit.client.1.smithi169.stdout:2/78: truncate d0/d5/fe 3356717 0 2024-09-10T03:42:30.834 INFO:tasks.workunit.client.1.smithi169.stdout:2/79: readlink d0/d5/l10 0 2024-09-10T03:42:30.836 INFO:tasks.workunit.client.1.smithi169.stdout:6/58: mkdir d0/d6/d18 0 2024-09-10T03:42:30.836 INFO:tasks.workunit.client.1.smithi169.stdout:6/59: write d0/fe [322370,87563] 0 2024-09-10T03:42:30.837 INFO:tasks.workunit.client.1.smithi169.stdout:6/60: write d0/f9 [9263390,114249] 0 2024-09-10T03:42:30.837 INFO:tasks.workunit.client.1.smithi169.stdout:6/61: chown d0/fc 13197 1 2024-09-10T03:42:30.837 INFO:tasks.workunit.client.1.smithi169.stdout:9/42: creat db/ff x:0 0 0 2024-09-10T03:42:30.837 INFO:tasks.workunit.client.1.smithi169.stdout:9/43: write db/fc [759645,70987] 0 2024-09-10T03:42:30.839 INFO:tasks.workunit.client.1.smithi169.stdout:0/101: creat d5/df/f26 x:0 0 0 2024-09-10T03:42:30.840 INFO:tasks.workunit.client.1.smithi169.stdout:4/55: creat d0/d6/f13 x:0 0 0 2024-09-10T03:42:30.842 INFO:tasks.workunit.client.1.smithi169.stdout:4/56: dread d0/d2/f5 [0,4194304] 0 2024-09-10T03:42:30.848 INFO:tasks.workunit.client.1.smithi169.stdout:9/44: mkdir db/d10 0 2024-09-10T03:42:30.848 INFO:tasks.workunit.client.1.smithi169.stdout:9/45: getdents db/d10 0 2024-09-10T03:42:30.848 INFO:tasks.workunit.client.1.smithi169.stdout:9/46: truncate db/ff 1013243 0 2024-09-10T03:42:30.849 INFO:tasks.workunit.client.1.smithi169.stdout:0/102: link d5/df/d1b/c1f d5/db/d24/c27 0 2024-09-10T03:42:30.849 INFO:tasks.workunit.client.1.smithi169.stdout:4/57: rmdir d0 39 2024-09-10T03:42:30.852 INFO:tasks.workunit.client.1.smithi169.stdout:0/103: creat d5/df/d13/f28 x:0 0 0 2024-09-10T03:42:30.855 INFO:tasks.workunit.client.1.smithi169.stdout:6/62: dwrite d0/d6/fb [0,4194304] 0 2024-09-10T03:42:30.855 INFO:tasks.workunit.client.1.smithi169.stdout:6/63: truncate d0/fc 4431410 0 2024-09-10T03:42:30.855 INFO:tasks.workunit.client.1.smithi169.stdout:4/58: creat d0/d2/f14 x:0 0 0 2024-09-10T03:42:30.855 INFO:tasks.workunit.client.1.smithi169.stdout:4/59: write d0/d2/f14 [348724,42668] 0 2024-09-10T03:42:30.856 INFO:tasks.workunit.client.1.smithi169.stdout:0/104: symlink d5/db/l29 0 2024-09-10T03:42:30.856 INFO:tasks.workunit.client.1.smithi169.stdout:0/105: stat d5/df 0 2024-09-10T03:42:30.856 INFO:tasks.workunit.client.1.smithi169.stdout:0/106: readlink d5/df/l1a 0 2024-09-10T03:42:30.856 INFO:tasks.workunit.client.1.smithi169.stdout:0/107: dread - d5/db/f25 zero size 2024-09-10T03:42:30.856 INFO:tasks.workunit.client.1.smithi169.stdout:4/60: write d0/d6/d7/fc [185530,32745] 0 2024-09-10T03:42:30.863 INFO:tasks.workunit.client.1.smithi169.stdout:0/108: rename d5/db/l29 to d5/db/d20/l2a 0 2024-09-10T03:42:30.866 INFO:tasks.workunit.client.1.smithi169.stdout:4/61: symlink d0/d6/l15 0 2024-09-10T03:42:30.869 INFO:tasks.workunit.client.1.smithi169.stdout:4/62: read d0/d6/d7/fc [1518950,129558] 0 2024-09-10T03:42:30.871 INFO:tasks.workunit.client.1.smithi169.stdout:0/109: creat d5/db/f2b x:0 0 0 2024-09-10T03:42:30.884 INFO:tasks.workunit.client.1.smithi169.stdout:0/110: write d5/df/d13/f19 [3911287,61289] 0 2024-09-10T03:42:30.887 INFO:tasks.workunit.client.1.smithi169.stdout:0/111: mkdir d5/df/d2c 0 2024-09-10T03:42:30.888 INFO:tasks.workunit.client.1.smithi169.stdout:4/63: dwrite d0/d6/f13 [0,4194304] 0 2024-09-10T03:42:30.894 INFO:tasks.workunit.client.1.smithi169.stdout:0/112: dread d5/df/d13/f19 [0,4194304] 0 2024-09-10T03:42:30.895 INFO:tasks.workunit.client.1.smithi169.stdout:0/113: read d5/df/d13/f19 [2485582,52477] 0 2024-09-10T03:42:30.896 INFO:tasks.workunit.client.1.smithi169.stdout:4/64: mkdir d0/d16 0 2024-09-10T03:42:30.896 INFO:tasks.workunit.client.1.smithi169.stdout:4/65: stat d0/d2 0 2024-09-10T03:42:30.899 INFO:tasks.workunit.client.1.smithi169.stdout:4/66: creat d0/d6/f17 x:0 0 0 2024-09-10T03:42:30.899 INFO:tasks.workunit.client.1.smithi169.stdout:4/67: read - d0/d2/fe zero size 2024-09-10T03:42:30.902 INFO:tasks.workunit.client.1.smithi169.stdout:4/68: mknod d0/d2/c18 0 2024-09-10T03:42:30.902 INFO:tasks.workunit.client.1.smithi169.stdout:4/69: stat d0/d6/d7/fc 0 2024-09-10T03:42:30.903 INFO:tasks.workunit.client.1.smithi169.stdout:4/70: mknod d0/d6/d7/c19 0 2024-09-10T03:42:30.908 INFO:tasks.workunit.client.1.smithi169.stdout:0/114: dwrite d5/fe [0,4194304] 0 2024-09-10T03:42:30.913 INFO:tasks.workunit.client.1.smithi169.stdout:0/115: mkdir d5/db/d2d 0 2024-09-10T03:42:30.913 INFO:tasks.workunit.client.1.smithi169.stdout:0/116: write d5/df/f14 [680124,108533] 0 2024-09-10T03:42:30.913 INFO:tasks.workunit.client.1.smithi169.stdout:0/117: chown d5 63036 1 2024-09-10T03:42:30.915 INFO:tasks.workunit.client.1.smithi169.stdout:0/118: rename d5/df/c22 to d5/df/d1b/c2e 0 2024-09-10T03:42:30.917 INFO:tasks.workunit.client.1.smithi169.stdout:0/119: creat d5/db/d2d/f2f x:0 0 0 2024-09-10T03:42:30.917 INFO:tasks.workunit.client.1.smithi169.stdout:4/71: dwrite d0/d2/f9 [0,4194304] 0 2024-09-10T03:42:30.919 INFO:tasks.workunit.client.1.smithi169.stdout:0/120: creat d5/db/d2d/f30 x:0 0 0 2024-09-10T03:42:30.920 INFO:tasks.workunit.client.1.smithi169.stdout:4/72: creat d0/d6/f1a x:0 0 0 2024-09-10T03:42:30.923 INFO:tasks.workunit.client.1.smithi169.stdout:0/121: mknod d5/df/c31 0 2024-09-10T03:42:30.924 INFO:tasks.workunit.client.1.smithi169.stdout:4/73: symlink d0/d6/l1b 0 2024-09-10T03:42:30.972 INFO:tasks.workunit.client.1.smithi169.stdout:8/53: sync 2024-09-10T03:42:30.972 INFO:tasks.workunit.client.1.smithi169.stdout:7/40: sync 2024-09-10T03:42:30.973 INFO:tasks.workunit.client.1.smithi169.stdout:1/51: sync 2024-09-10T03:42:30.974 INFO:tasks.workunit.client.1.smithi169.stdout:1/52: creat ff x:0 0 0 2024-09-10T03:42:30.974 INFO:tasks.workunit.client.1.smithi169.stdout:1/53: fdatasync f8 0 2024-09-10T03:42:30.974 INFO:tasks.workunit.client.1.smithi169.stdout:1/54: chown l6 454 1 2024-09-10T03:42:30.974 INFO:tasks.workunit.client.1.smithi169.stdout:1/55: dread - ff zero size 2024-09-10T03:42:30.983 INFO:tasks.workunit.client.1.smithi169.stdout:7/41: write d0/f5 [3127925,13400] 0 2024-09-10T03:42:30.987 INFO:tasks.workunit.client.1.smithi169.stdout:7/42: mkdir d0/dd 0 2024-09-10T03:42:30.989 INFO:tasks.workunit.client.1.smithi169.stdout:7/43: creat d0/fe x:0 0 0 2024-09-10T03:42:30.989 INFO:tasks.workunit.client.1.smithi169.stdout:2/80: fsync d0/d5/fe 0 2024-09-10T03:42:30.991 INFO:tasks.workunit.client.1.smithi169.stdout:7/44: write d0/f2 [507893,64619] 0 2024-09-10T03:42:30.991 INFO:tasks.workunit.client.1.smithi169.stdout:7/45: dread - d0/fe zero size 2024-09-10T03:42:30.994 INFO:tasks.workunit.client.1.smithi169.stdout:7/46: link d0/l7 d0/lf 0 2024-09-10T03:42:30.996 INFO:tasks.workunit.client.1.smithi169.stdout:7/47: mkdir d0/d10 0 2024-09-10T03:42:30.996 INFO:tasks.workunit.client.1.smithi169.stdout:7/48: stat d0/fe 0 2024-09-10T03:42:30.998 INFO:tasks.workunit.client.1.smithi169.stdout:8/54: dwrite d3/de/fb [0,4194304] 0 2024-09-10T03:42:30.998 INFO:tasks.workunit.client.1.smithi169.stdout:8/55: chown d3/dc/f11 175 1 2024-09-10T03:42:30.998 INFO:tasks.workunit.client.1.smithi169.stdout:1/56: dwrite fa [0,4194304] 0 2024-09-10T03:42:30.998 INFO:tasks.workunit.client.1.smithi169.stdout:1/57: chown f3 804708 1 2024-09-10T03:42:30.998 INFO:tasks.workunit.client.1.smithi169.stdout:1/58: chown l6 77962974 1 2024-09-10T03:42:30.998 INFO:tasks.workunit.client.1.smithi169.stdout:1/59: chown lc 12 1 2024-09-10T03:42:30.999 INFO:tasks.workunit.client.1.smithi169.stdout:7/49: symlink d0/dd/l11 0 2024-09-10T03:42:31.002 INFO:tasks.workunit.client.1.smithi169.stdout:8/56: dread d3/dc/f11 [0,4194304] 0 2024-09-10T03:42:31.003 INFO:tasks.workunit.client.1.smithi169.stdout:9/47: getdents db 0 2024-09-10T03:42:31.008 INFO:tasks.workunit.client.1.smithi169.stdout:1/60: symlink l10 0 2024-09-10T03:42:31.008 INFO:tasks.workunit.client.1.smithi169.stdout:1/61: truncate f7 1533737 0 2024-09-10T03:42:31.012 INFO:tasks.workunit.client.1.smithi169.stdout:8/57: mknod d3/de/c13 0 2024-09-10T03:42:31.012 INFO:tasks.workunit.client.1.smithi169.stdout:8/58: fsync d3/de/fb 0 2024-09-10T03:42:31.012 INFO:tasks.workunit.client.1.smithi169.stdout:8/59: chown d3/f7 22925 1 2024-09-10T03:42:31.013 INFO:tasks.workunit.client.1.smithi169.stdout:4/74: fsync d0/d2/f14 0 2024-09-10T03:42:31.013 INFO:tasks.workunit.client.1.smithi169.stdout:4/75: truncate d0/d6/fa 362433 0 2024-09-10T03:42:31.014 INFO:tasks.workunit.client.1.smithi169.stdout:1/62: symlink l11 0 2024-09-10T03:42:31.015 INFO:tasks.workunit.client.1.smithi169.stdout:8/60: rename d3/f7 to d3/de/f14 0 2024-09-10T03:42:31.016 INFO:tasks.workunit.client.1.smithi169.stdout:4/76: rename d0/d16 to d0/d16/d1c 22 2024-09-10T03:42:31.017 INFO:tasks.workunit.client.1.smithi169.stdout:6/64: truncate d0/d6/fb 1992221 0 2024-09-10T03:42:31.017 INFO:tasks.workunit.client.1.smithi169.stdout:6/65: chown d0/d6 1747688 1 2024-09-10T03:42:31.018 INFO:tasks.workunit.client.1.smithi169.stdout:1/63: unlink le 0 2024-09-10T03:42:31.018 INFO:tasks.workunit.client.1.smithi169.stdout:1/64: readlink l6 0 2024-09-10T03:42:31.020 INFO:tasks.workunit.client.1.smithi169.stdout:8/61: unlink d3/dc/f11 0 2024-09-10T03:42:31.020 INFO:tasks.workunit.client.1.smithi169.stdout:8/62: stat d3/de/l12 0 2024-09-10T03:42:31.023 INFO:tasks.workunit.client.1.smithi169.stdout:6/66: getdents d0/d10 0 2024-09-10T03:42:31.024 INFO:tasks.workunit.client.1.smithi169.stdout:9/48: dwrite db/fe [0,4194304] 0 2024-09-10T03:42:31.025 INFO:tasks.workunit.client.1.smithi169.stdout:7/50: dwrite d0/f5 [4194304,4194304] 0 2024-09-10T03:42:31.026 INFO:tasks.workunit.client.1.smithi169.stdout:6/67: creat d0/d10/f19 x:0 0 0 2024-09-10T03:42:31.027 INFO:tasks.workunit.client.1.smithi169.stdout:6/68: getdents d0/d12 0 2024-09-10T03:42:31.030 INFO:tasks.workunit.client.1.smithi169.stdout:9/49: creat db/d10/f11 x:0 0 0 2024-09-10T03:42:31.030 INFO:tasks.workunit.client.1.smithi169.stdout:1/65: write fa [3861955,61373] 0 2024-09-10T03:42:31.030 INFO:tasks.workunit.client.1.smithi169.stdout:1/66: readlink l6 0 2024-09-10T03:42:31.040 INFO:tasks.workunit.client.1.smithi169.stdout:8/63: dwrite f2 [0,4194304] 0 2024-09-10T03:42:31.040 INFO:tasks.workunit.client.1.smithi169.stdout:6/69: dread d0/f9 [4194304,4194304] 0 2024-09-10T03:42:31.040 INFO:tasks.workunit.client.1.smithi169.stdout:7/51: creat d0/dd/f12 x:0 0 0 2024-09-10T03:42:31.040 INFO:tasks.workunit.client.1.smithi169.stdout:6/70: truncate d0/f8 896211 0 2024-09-10T03:42:31.040 INFO:tasks.workunit.client.1.smithi169.stdout:6/71: fsync d0/fe 0 2024-09-10T03:42:31.040 INFO:tasks.workunit.client.1.smithi169.stdout:7/52: dread - d0/dd/f12 zero size 2024-09-10T03:42:31.040 INFO:tasks.workunit.client.1.smithi169.stdout:6/72: write d0/f17 [74119,117004] 0 2024-09-10T03:42:31.040 INFO:tasks.workunit.client.1.smithi169.stdout:4/77: dwrite d0/d6/d7/f10 [0,4194304] 0 2024-09-10T03:42:31.040 INFO:tasks.workunit.client.1.smithi169.stdout:4/78: dread - d0/d2/fe zero size 2024-09-10T03:42:31.040 INFO:tasks.workunit.client.1.smithi169.stdout:4/79: readlink d0/d6/l1b 0 2024-09-10T03:42:31.040 INFO:tasks.workunit.client.1.smithi169.stdout:6/73: write d0/f8 [605813,7827] 0 2024-09-10T03:42:31.041 INFO:tasks.workunit.client.1.smithi169.stdout:9/50: rename c3 to db/c12 0 2024-09-10T03:42:31.043 INFO:tasks.workunit.client.1.smithi169.stdout:1/67: mkdir d12 0 2024-09-10T03:42:31.044 INFO:tasks.workunit.client.1.smithi169.stdout:1/68: read - f8 zero size 2024-09-10T03:42:31.050 INFO:tasks.workunit.client.1.smithi169.stdout:8/64: link f2 d3/de/f15 0 2024-09-10T03:42:31.050 INFO:tasks.workunit.client.1.smithi169.stdout:8/65: chown d3/dc 201 1 2024-09-10T03:42:31.050 INFO:tasks.workunit.client.1.smithi169.stdout:8/66: fdatasync d3/de/f15 0 2024-09-10T03:42:31.050 INFO:tasks.workunit.client.1.smithi169.stdout:8/67: chown d3/de/f14 25 1 2024-09-10T03:42:31.053 INFO:tasks.workunit.client.1.smithi169.stdout:4/80: mkdir d0/d1d 0 2024-09-10T03:42:31.053 INFO:tasks.workunit.client.1.smithi169.stdout:4/81: chown d0/d16 1296 1 2024-09-10T03:42:31.056 INFO:tasks.workunit.client.1.smithi169.stdout:9/51: creat db/f13 x:0 0 0 2024-09-10T03:42:31.057 INFO:tasks.workunit.client.1.smithi169.stdout:1/69: symlink d12/l13 0 2024-09-10T03:42:31.059 INFO:tasks.workunit.client.1.smithi169.stdout:9/52: unlink l9 0 2024-09-10T03:42:31.059 INFO:tasks.workunit.client.1.smithi169.stdout:9/53: stat db 0 2024-09-10T03:42:31.061 INFO:tasks.workunit.client.1.smithi169.stdout:1/70: symlink d12/l14 0 2024-09-10T03:42:31.061 INFO:tasks.workunit.client.1.smithi169.stdout:1/71: dread - f8 zero size 2024-09-10T03:42:31.062 INFO:tasks.workunit.client.1.smithi169.stdout:1/72: rmdir d12 39 2024-09-10T03:42:31.062 INFO:tasks.workunit.client.1.smithi169.stdout:1/73: chown d12 42810 1 2024-09-10T03:42:31.063 INFO:tasks.workunit.client.1.smithi169.stdout:7/53: dwrite d0/fe [0,4194304] 0 2024-09-10T03:42:31.063 INFO:tasks.workunit.client.1.smithi169.stdout:7/54: chown d0/lf 0 1 2024-09-10T03:42:31.064 INFO:tasks.workunit.client.1.smithi169.stdout:8/68: write d3/de/fb [2164778,67569] 0 2024-09-10T03:42:31.064 INFO:tasks.workunit.client.1.smithi169.stdout:8/69: truncate d3/f4 1684525 0 2024-09-10T03:42:31.065 INFO:tasks.workunit.client.1.smithi169.stdout:6/74: dwrite d0/d10/f19 [0,4194304] 0 2024-09-10T03:42:31.065 INFO:tasks.workunit.client.1.smithi169.stdout:1/74: creat d12/f15 x:0 0 0 2024-09-10T03:42:31.065 INFO:tasks.workunit.client.1.smithi169.stdout:1/75: fdatasync f7 0 2024-09-10T03:42:31.072 INFO:tasks.workunit.client.1.smithi169.stdout:7/55: symlink d0/l13 0 2024-09-10T03:42:31.072 INFO:tasks.workunit.client.1.smithi169.stdout:7/56: write d0/f3 [2039964,77482] 0 2024-09-10T03:42:31.076 INFO:tasks.workunit.client.1.smithi169.stdout:9/54: dwrite f7 [4194304,4194304] 0 2024-09-10T03:42:31.078 INFO:tasks.workunit.client.1.smithi169.stdout:8/70: rmdir d3 39 2024-09-10T03:42:31.081 INFO:tasks.workunit.client.1.smithi169.stdout:1/76: creat d12/f16 x:0 0 0 2024-09-10T03:42:31.081 INFO:tasks.workunit.client.1.smithi169.stdout:1/77: stat f5 0 2024-09-10T03:42:31.081 INFO:tasks.workunit.client.1.smithi169.stdout:1/78: truncate d12/f16 736864 0 2024-09-10T03:42:31.087 INFO:tasks.workunit.client.1.smithi169.stdout:6/75: link d0/c1 d0/d10/c1a 0 2024-09-10T03:42:31.087 INFO:tasks.workunit.client.1.smithi169.stdout:6/76: write d0/d6/f13 [1095094,39319] 0 2024-09-10T03:42:31.088 INFO:tasks.workunit.client.1.smithi169.stdout:8/71: dread f2 [0,4194304] 0 2024-09-10T03:42:31.088 INFO:tasks.workunit.client.1.smithi169.stdout:8/72: readlink d3/l6 0 2024-09-10T03:42:31.088 INFO:tasks.workunit.client.1.smithi169.stdout:8/73: read d3/de/fb [2085453,16035] 0 2024-09-10T03:42:31.089 INFO:tasks.workunit.client.1.smithi169.stdout:8/74: dread - d3/de/f14 zero size 2024-09-10T03:42:31.089 INFO:tasks.workunit.client.1.smithi169.stdout:8/75: read d3/f4 [219354,112102] 0 2024-09-10T03:42:31.091 INFO:tasks.workunit.client.1.smithi169.stdout:9/55: dwrite db/fe [0,4194304] 0 2024-09-10T03:42:31.096 INFO:tasks.workunit.client.1.smithi169.stdout:6/77: dread d0/fe [0,4194304] 0 2024-09-10T03:42:31.096 INFO:tasks.workunit.client.1.smithi169.stdout:6/78: chown d0/d6/fb 0 1 2024-09-10T03:42:31.096 INFO:tasks.workunit.client.1.smithi169.stdout:6/79: chown d0/l2 507159624 1 2024-09-10T03:42:31.096 INFO:tasks.workunit.client.1.smithi169.stdout:6/80: read d0/f17 [57877,31567] 0 2024-09-10T03:42:31.099 INFO:tasks.workunit.client.1.smithi169.stdout:6/81: creat d0/d6/f1b x:0 0 0 2024-09-10T03:42:31.099 INFO:tasks.workunit.client.1.smithi169.stdout:6/82: chown d0/d12 6479993 1 2024-09-10T03:42:31.101 INFO:tasks.workunit.client.1.smithi169.stdout:6/83: creat d0/d12/f1c x:0 0 0 2024-09-10T03:42:31.103 INFO:tasks.workunit.client.1.smithi169.stdout:6/84: creat d0/d10/f1d x:0 0 0 2024-09-10T03:42:31.105 INFO:tasks.workunit.client.1.smithi169.stdout:9/56: dwrite f7 [0,4194304] 0 2024-09-10T03:42:31.121 INFO:tasks.workunit.client.1.smithi169.stdout:9/57: dwrite db/f13 [0,4194304] 0 2024-09-10T03:42:31.128 INFO:tasks.workunit.client.1.smithi169.stdout:9/58: dread f7 [0,4194304] 0 2024-09-10T03:42:31.128 INFO:tasks.workunit.client.1.smithi169.stdout:9/59: chown db/c12 21777115 1 2024-09-10T03:42:31.137 INFO:tasks.workunit.client.1.smithi169.stdout:9/60: dread db/f13 [0,4194304] 0 2024-09-10T03:42:31.137 INFO:tasks.workunit.client.1.smithi169.stdout:9/61: stat c6 0 2024-09-10T03:42:31.151 INFO:tasks.workunit.client.1.smithi169.stdout:9/62: dwrite db/d10/f11 [0,4194304] 0 2024-09-10T03:42:31.156 INFO:tasks.workunit.client.1.smithi169.stdout:5/63: sync 2024-09-10T03:42:31.156 INFO:tasks.workunit.client.1.smithi169.stdout:3/81: sync 2024-09-10T03:42:31.156 INFO:tasks.workunit.client.1.smithi169.stdout:5/64: chown d3/de 0 1 2024-09-10T03:42:31.156 INFO:tasks.workunit.client.1.smithi169.stdout:5/65: stat d3/da/df/c10 0 2024-09-10T03:42:31.157 INFO:tasks.workunit.client.1.smithi169.stdout:9/63: rename c0 to db/d10/c14 0 2024-09-10T03:42:31.161 INFO:tasks.workunit.client.1.smithi169.stdout:3/82: unlink d0/dc/f12 0 2024-09-10T03:42:31.161 INFO:tasks.workunit.client.1.smithi169.stdout:3/83: write d0/dc/f14 [988480,34211] 0 2024-09-10T03:42:31.161 INFO:tasks.workunit.client.1.smithi169.stdout:3/84: write d0/dc/f15 [717580,37054] 0 2024-09-10T03:42:31.161 INFO:tasks.workunit.client.1.smithi169.stdout:3/85: write d0/fa [3190719,113279] 0 2024-09-10T03:42:31.162 INFO:tasks.workunit.client.1.smithi169.stdout:6/85: dread d0/d6/fb [0,4194304] 0 2024-09-10T03:42:31.162 INFO:tasks.workunit.client.1.smithi169.stdout:6/86: write d0/f17 [612065,92378] 0 2024-09-10T03:42:31.163 INFO:tasks.workunit.client.1.smithi169.stdout:5/66: link d3/da/fb d3/f12 0 2024-09-10T03:42:31.163 INFO:tasks.workunit.client.1.smithi169.stdout:6/87: write d0/d6/f1b [555361,130358] 0 2024-09-10T03:42:31.164 INFO:tasks.workunit.client.1.smithi169.stdout:6/88: write d0/d10/f19 [530924,77820] 0 2024-09-10T03:42:31.165 INFO:tasks.workunit.client.1.smithi169.stdout:9/64: link db/ld db/l15 0 2024-09-10T03:42:31.168 INFO:tasks.workunit.client.1.smithi169.stdout:0/122: sync 2024-09-10T03:42:31.168 INFO:tasks.workunit.client.1.smithi169.stdout:3/86: rmdir d0 39 2024-09-10T03:42:31.170 INFO:tasks.workunit.client.1.smithi169.stdout:5/67: mkdir d3/da/df/d13 0 2024-09-10T03:42:31.180 INFO:tasks.workunit.client.1.smithi169.stdout:0/123: rename d5/df/d13/f16 to d5/df/f32 0 2024-09-10T03:42:31.181 INFO:tasks.workunit.client.1.smithi169.stdout:0/124: fsync d5/df/d13/f1d 0 2024-09-10T03:42:31.181 INFO:tasks.workunit.client.1.smithi169.stdout:9/65: dwrite db/d10/f11 [0,4194304] 0 2024-09-10T03:42:31.181 INFO:tasks.workunit.client.1.smithi169.stdout:3/87: link d0/dc/l18 d0/dc/l19 0 2024-09-10T03:42:31.181 INFO:tasks.workunit.client.1.smithi169.stdout:3/88: readlink d0/l8 0 2024-09-10T03:42:31.184 INFO:tasks.workunit.client.1.smithi169.stdout:8/76: fsync f2 0 2024-09-10T03:42:31.187 INFO:tasks.workunit.client.1.smithi169.stdout:5/68: rename d3/de to d3/da/df/d14 0 2024-09-10T03:42:31.192 INFO:tasks.workunit.client.1.smithi169.stdout:0/125: mkdir d5/df/d1b/d33 0 2024-09-10T03:42:31.195 INFO:tasks.workunit.client.1.smithi169.stdout:9/66: creat db/f16 x:0 0 0 2024-09-10T03:42:31.197 INFO:tasks.workunit.client.1.smithi169.stdout:5/69: symlink d3/l15 0 2024-09-10T03:42:31.198 INFO:tasks.workunit.client.1.smithi169.stdout:0/126: mknod d5/db/d2d/c34 0 2024-09-10T03:42:31.201 INFO:tasks.workunit.client.1.smithi169.stdout:9/67: mknod db/c17 0 2024-09-10T03:42:31.201 INFO:tasks.workunit.client.1.smithi169.stdout:5/70: truncate d3/f12 4073431 0 2024-09-10T03:42:31.202 INFO:tasks.workunit.client.1.smithi169.stdout:8/77: dwrite f2 [0,4194304] 0 2024-09-10T03:42:31.202 INFO:tasks.workunit.client.1.smithi169.stdout:8/78: chown d3/ca 204 1 2024-09-10T03:42:31.202 INFO:tasks.workunit.client.1.smithi169.stdout:3/89: dwrite d0/dc/fe [0,4194304] 0 2024-09-10T03:42:31.203 INFO:tasks.workunit.client.1.smithi169.stdout:0/127: mknod d5/db/d20/c35 0 2024-09-10T03:42:31.204 INFO:tasks.workunit.client.1.smithi169.stdout:9/68: mkdir db/d10/d18 0 2024-09-10T03:42:31.207 INFO:tasks.workunit.client.1.smithi169.stdout:8/79: dread d3/de/fb [0,4194304] 0 2024-09-10T03:42:31.208 INFO:tasks.workunit.client.1.smithi169.stdout:3/90: creat d0/f1a x:0 0 0 2024-09-10T03:42:31.208 INFO:tasks.workunit.client.1.smithi169.stdout:3/91: stat d0/l5 0 2024-09-10T03:42:31.208 INFO:tasks.workunit.client.1.smithi169.stdout:3/92: fdatasync d0/dc/f15 0 2024-09-10T03:42:31.208 INFO:tasks.workunit.client.1.smithi169.stdout:0/128: creat d5/db/d2d/f36 x:0 0 0 2024-09-10T03:42:31.210 INFO:tasks.workunit.client.1.smithi169.stdout:9/69: unlink db/f16 0 2024-09-10T03:42:31.211 INFO:tasks.workunit.client.1.smithi169.stdout:3/93: unlink d0/cb 0 2024-09-10T03:42:31.211 INFO:tasks.workunit.client.1.smithi169.stdout:3/94: getdents d0/dc/d17 0 2024-09-10T03:42:31.214 INFO:tasks.workunit.client.1.smithi169.stdout:9/70: chown l4 4088131 1 2024-09-10T03:42:31.214 INFO:tasks.workunit.client.1.smithi169.stdout:9/71: getdents db/d10/d18 0 2024-09-10T03:42:31.218 INFO:tasks.workunit.client.1.smithi169.stdout:8/80: read f2 [4046677,16557] 0 2024-09-10T03:42:31.218 INFO:tasks.workunit.client.1.smithi169.stdout:7/57: truncate d0/f2 1678303 0 2024-09-10T03:42:31.220 INFO:tasks.workunit.client.1.smithi169.stdout:8/81: rmdir d3/de 39 2024-09-10T03:42:31.222 INFO:tasks.workunit.client.1.smithi169.stdout:7/58: rmdir d0/dd 39 2024-09-10T03:42:31.226 INFO:tasks.workunit.client.1.smithi169.stdout:0/129: dwrite d5/df/f26 [0,4194304] 0 2024-09-10T03:42:31.226 INFO:tasks.workunit.client.1.smithi169.stdout:0/130: dread - d5/db/f25 zero size 2024-09-10T03:42:31.227 INFO:tasks.workunit.client.1.smithi169.stdout:0/131: fdatasync d5/fe 0 2024-09-10T03:42:31.227 INFO:tasks.workunit.client.1.smithi169.stdout:9/72: dread db/fc [0,4194304] 0 2024-09-10T03:42:31.243 INFO:tasks.workunit.client.1.smithi169.stdout:9/73: read f7 [7242824,107532] 0 2024-09-10T03:42:31.244 INFO:tasks.workunit.client.1.smithi169.stdout:3/95: dwrite d0/dc/f15 [0,4194304] 0 2024-09-10T03:42:31.245 INFO:tasks.workunit.client.1.smithi169.stdout:3/96: chown d0/c9 237653352 1 2024-09-10T03:42:31.245 INFO:tasks.workunit.client.1.smithi169.stdout:9/74: creat db/f19 x:0 0 0 2024-09-10T03:42:31.249 INFO:tasks.workunit.client.1.smithi169.stdout:3/97: link d0/dc/cf d0/dc/d17/c1b 0 2024-09-10T03:42:31.255 INFO:tasks.workunit.client.1.smithi169.stdout:3/98: write d0/f4 [887704,108136] 0 2024-09-10T03:42:31.255 INFO:tasks.workunit.client.1.smithi169.stdout:3/99: fsync d0/f13 0 2024-09-10T03:42:31.257 INFO:tasks.workunit.client.1.smithi169.stdout:3/100: mkdir d0/dc/d17/d1c 0 2024-09-10T03:42:31.257 INFO:tasks.workunit.client.1.smithi169.stdout:3/101: readlink d0/dc/l19 0 2024-09-10T03:42:31.257 INFO:tasks.workunit.client.1.smithi169.stdout:3/102: stat d0/dc/f15 0 2024-09-10T03:42:31.257 INFO:tasks.workunit.client.1.smithi169.stdout:8/82: dwrite f2 [0,4194304] 0 2024-09-10T03:42:31.261 INFO:tasks.workunit.client.1.smithi169.stdout:7/59: dwrite d0/fa [0,4194304] 0 2024-09-10T03:42:31.262 INFO:tasks.workunit.client.1.smithi169.stdout:0/132: dwrite d5/db/d2d/f2f [0,4194304] 0 2024-09-10T03:42:31.263 INFO:tasks.workunit.client.1.smithi169.stdout:9/75: dwrite f7 [0,4194304] 0 2024-09-10T03:42:31.263 INFO:tasks.workunit.client.1.smithi169.stdout:8/83: getdents d3/de 0 2024-09-10T03:42:31.264 INFO:tasks.workunit.client.1.smithi169.stdout:8/84: chown d3/de/f15 5152 1 2024-09-10T03:42:31.269 INFO:tasks.workunit.client.1.smithi169.stdout:9/76: write db/fc [2933719,64949] 0 2024-09-10T03:42:31.269 INFO:tasks.workunit.client.1.smithi169.stdout:9/77: fsync f7 0 2024-09-10T03:42:31.269 INFO:tasks.workunit.client.1.smithi169.stdout:9/78: chown db/fc 344 1 2024-09-10T03:42:31.272 INFO:tasks.workunit.client.1.smithi169.stdout:0/133: dread f4 [4194304,4194304] 0 2024-09-10T03:42:31.272 INFO:tasks.workunit.client.1.smithi169.stdout:9/79: read db/f13 [1054517,123326] 0 2024-09-10T03:42:31.272 INFO:tasks.workunit.client.1.smithi169.stdout:9/80: chown db/f19 31 1 2024-09-10T03:42:31.272 INFO:tasks.workunit.client.1.smithi169.stdout:9/81: write db/f13 [965450,28068] 0 2024-09-10T03:42:31.272 INFO:tasks.workunit.client.1.smithi169.stdout:0/134: write d5/db/d2d/f30 [877083,47482] 0 2024-09-10T03:42:31.289 INFO:tasks.workunit.client.1.smithi169.stdout:7/60: dwrite d0/dd/f12 [0,4194304] 0 2024-09-10T03:42:31.290 INFO:tasks.workunit.client.1.smithi169.stdout:7/61: stat d0/f2 0 2024-09-10T03:42:31.293 INFO:tasks.workunit.client.1.smithi169.stdout:8/85: dwrite f2 [0,4194304] 0 2024-09-10T03:42:31.293 INFO:tasks.workunit.client.1.smithi169.stdout:9/82: dwrite db/ff [0,4194304] 0 2024-09-10T03:42:31.293 INFO:tasks.workunit.client.1.smithi169.stdout:7/62: unlink d0/fe 0 2024-09-10T03:42:31.296 INFO:tasks.workunit.client.1.smithi169.stdout:8/86: dread d3/de/fb [0,4194304] 0 2024-09-10T03:42:31.302 INFO:tasks.workunit.client.1.smithi169.stdout:7/63: write d0/fa [3323405,42053] 0 2024-09-10T03:42:31.303 INFO:tasks.workunit.client.1.smithi169.stdout:7/64: chown d0/d10 23291 1 2024-09-10T03:42:31.303 INFO:tasks.workunit.client.1.smithi169.stdout:9/83: mknod db/d10/d18/c1a 0 2024-09-10T03:42:31.303 INFO:tasks.workunit.client.1.smithi169.stdout:9/84: write db/f13 [731980,83491] 0 2024-09-10T03:42:31.306 INFO:tasks.workunit.client.1.smithi169.stdout:8/87: unlink d3/de/c9 0 2024-09-10T03:42:31.307 INFO:tasks.workunit.client.1.smithi169.stdout:8/88: truncate f2 4374802 0 2024-09-10T03:42:31.307 INFO:tasks.workunit.client.1.smithi169.stdout:9/85: rmdir db 39 2024-09-10T03:42:31.307 INFO:tasks.workunit.client.1.smithi169.stdout:9/86: write db/f19 [276619,81613] 0 2024-09-10T03:42:31.307 INFO:tasks.workunit.client.1.smithi169.stdout:9/87: write db/f19 [575585,112299] 0 2024-09-10T03:42:31.309 INFO:tasks.workunit.client.1.smithi169.stdout:8/89: creat d3/de/f16 x:0 0 0 2024-09-10T03:42:31.311 INFO:tasks.workunit.client.1.smithi169.stdout:9/88: mkdir db/d10/d18/d1b 0 2024-09-10T03:42:31.317 INFO:tasks.workunit.client.1.smithi169.stdout:0/135: rename d5/df/f32 to d5/df/d1b/d33/f37 0 2024-09-10T03:42:31.317 INFO:tasks.workunit.client.1.smithi169.stdout:0/136: write d5/f7 [200592,97995] 0 2024-09-10T03:42:31.317 INFO:tasks.workunit.client.1.smithi169.stdout:0/137: write d5/db/f10 [401279,88345] 0 2024-09-10T03:42:31.318 INFO:tasks.workunit.client.1.smithi169.stdout:8/90: dread f2 [0,4194304] 0 2024-09-10T03:42:31.318 INFO:tasks.workunit.client.1.smithi169.stdout:8/91: chown d3/de/l12 2899940 1 2024-09-10T03:42:31.318 INFO:tasks.workunit.client.1.smithi169.stdout:8/92: readlink d3/l6 0 2024-09-10T03:42:31.321 INFO:tasks.workunit.client.1.smithi169.stdout:0/138: symlink d5/db/d20/l38 0 2024-09-10T03:42:31.323 INFO:tasks.workunit.client.1.smithi169.stdout:8/93: rmdir d3/de 39 2024-09-10T03:42:31.325 INFO:tasks.workunit.client.1.smithi169.stdout:0/139: symlink d5/df/d1b/l39 0 2024-09-10T03:42:31.325 INFO:tasks.workunit.client.1.smithi169.stdout:0/140: write d5/db/d2d/f36 [408973,129653] 0 2024-09-10T03:42:31.325 INFO:tasks.workunit.client.1.smithi169.stdout:0/141: readlink d5/df/l1a 0 2024-09-10T03:42:31.325 INFO:tasks.workunit.client.1.smithi169.stdout:0/142: read f4 [1215359,101071] 0 2024-09-10T03:42:31.327 INFO:tasks.workunit.client.1.smithi169.stdout:8/94: symlink d3/dc/l17 0 2024-09-10T03:42:31.327 INFO:tasks.workunit.client.1.smithi169.stdout:7/65: dwrite d0/fa [0,4194304] 0 2024-09-10T03:42:31.328 INFO:tasks.workunit.client.1.smithi169.stdout:0/143: mkdir d5/db/d2d/d3a 0 2024-09-10T03:42:31.336 INFO:tasks.workunit.client.1.smithi169.stdout:0/144: mknod d5/db/d24/c3b 0 2024-09-10T03:42:31.337 INFO:tasks.workunit.client.1.smithi169.stdout:7/66: dread d0/f5 [4194304,4194304] 0 2024-09-10T03:42:31.337 INFO:tasks.workunit.client.1.smithi169.stdout:7/67: stat d0/dd 0 2024-09-10T03:42:31.337 INFO:tasks.workunit.client.1.smithi169.stdout:7/68: fsync d0/fa 0 2024-09-10T03:42:31.339 INFO:tasks.workunit.client.1.smithi169.stdout:5/71: truncate d3/f12 4253594 0 2024-09-10T03:42:31.342 INFO:tasks.workunit.client.1.smithi169.stdout:7/69: symlink d0/d10/l14 0 2024-09-10T03:42:31.343 INFO:tasks.workunit.client.1.smithi169.stdout:0/145: dread d5/df/d13/f19 [0,4194304] 0 2024-09-10T03:42:31.344 INFO:tasks.workunit.client.1.smithi169.stdout:7/70: rmdir d0 39 2024-09-10T03:42:31.344 INFO:tasks.workunit.client.1.smithi169.stdout:7/71: chown d0/dd/f12 201956823 1 2024-09-10T03:42:31.344 INFO:tasks.workunit.client.1.smithi169.stdout:7/72: fdatasync d0/fa 0 2024-09-10T03:42:31.346 INFO:tasks.workunit.client.1.smithi169.stdout:0/146: truncate d5/df/f14 653123 0 2024-09-10T03:42:31.347 INFO:tasks.workunit.client.1.smithi169.stdout:7/73: creat d0/dd/f15 x:0 0 0 2024-09-10T03:42:31.350 INFO:tasks.workunit.client.1.smithi169.stdout:0/147: creat d5/df/d2c/f3c x:0 0 0 2024-09-10T03:42:31.350 INFO:tasks.workunit.client.1.smithi169.stdout:0/148: truncate d5/db/f25 458411 0 2024-09-10T03:42:31.351 INFO:tasks.workunit.client.1.smithi169.stdout:7/74: mkdir d0/d16 0 2024-09-10T03:42:31.351 INFO:tasks.workunit.client.1.smithi169.stdout:2/81: sync 2024-09-10T03:42:31.352 INFO:tasks.workunit.client.1.smithi169.stdout:4/82: sync 2024-09-10T03:42:31.353 INFO:tasks.workunit.client.1.smithi169.stdout:0/149: mknod d5/df/d13/c3d 0 2024-09-10T03:42:31.354 INFO:tasks.workunit.client.1.smithi169.stdout:7/75: mkdir d0/d10/d17 0 2024-09-10T03:42:31.356 INFO:tasks.workunit.client.1.smithi169.stdout:4/83: creat d0/d16/f1e x:0 0 0 2024-09-10T03:42:31.357 INFO:tasks.workunit.client.1.smithi169.stdout:7/76: dread d0/f2 [0,4194304] 0 2024-09-10T03:42:31.357 INFO:tasks.workunit.client.1.smithi169.stdout:7/77: fsync d0/f3 0 2024-09-10T03:42:31.360 INFO:tasks.workunit.client.1.smithi169.stdout:4/84: getdents d0/d6 0 2024-09-10T03:42:31.361 INFO:tasks.workunit.client.1.smithi169.stdout:0/150: dread d5/df/d13/f19 [0,4194304] 0 2024-09-10T03:42:31.362 INFO:tasks.workunit.client.1.smithi169.stdout:4/85: truncate d0/d6/d7/f10 267108 0 2024-09-10T03:42:31.364 INFO:tasks.workunit.client.1.smithi169.stdout:0/151: mknod d5/df/c3e 0 2024-09-10T03:42:31.364 INFO:tasks.workunit.client.1.smithi169.stdout:7/78: dread d0/dd/f12 [0,4194304] 0 2024-09-10T03:42:31.364 INFO:tasks.workunit.client.1.smithi169.stdout:7/79: stat d0 0 2024-09-10T03:42:31.364 INFO:tasks.workunit.client.1.smithi169.stdout:7/80: fdatasync d0/fa 0 2024-09-10T03:42:31.364 INFO:tasks.workunit.client.1.smithi169.stdout:7/81: chown d0 3 1 2024-09-10T03:42:31.366 INFO:tasks.workunit.client.1.smithi169.stdout:4/86: creat d0/d2/d11/f1f x:0 0 0 2024-09-10T03:42:31.368 INFO:tasks.workunit.client.1.smithi169.stdout:4/87: creat d0/d2/d11/f20 x:0 0 0 2024-09-10T03:42:31.369 INFO:tasks.workunit.client.1.smithi169.stdout:0/152: write f4 [6731051,114349] 0 2024-09-10T03:42:31.370 INFO:tasks.workunit.client.1.smithi169.stdout:2/82: dwrite d0/f15 [0,4194304] 0 2024-09-10T03:42:31.381 INFO:tasks.workunit.client.1.smithi169.stdout:0/153: dread f4 [0,4194304] 0 2024-09-10T03:42:31.382 INFO:tasks.workunit.client.1.smithi169.stdout:3/103: truncate d0/f4 4139929 0 2024-09-10T03:42:31.386 INFO:tasks.workunit.client.1.smithi169.stdout:2/83: write d0/f2 [2180415,25244] 0 2024-09-10T03:42:31.387 INFO:tasks.workunit.client.1.smithi169.stdout:3/104: rename d0/dc/d17/d1c to d0/d1d 0 2024-09-10T03:42:31.392 INFO:tasks.workunit.client.1.smithi169.stdout:2/84: creat d0/f18 x:0 0 0 2024-09-10T03:42:31.393 INFO:tasks.workunit.client.1.smithi169.stdout:3/105: creat d0/dc/d17/f1e x:0 0 0 2024-09-10T03:42:31.394 INFO:tasks.workunit.client.1.smithi169.stdout:0/154: read d5/df/d1b/d33/f37 [2065539,55213] 0 2024-09-10T03:42:31.396 INFO:tasks.workunit.client.1.smithi169.stdout:2/85: mkdir d0/d16/d19 0 2024-09-10T03:42:31.396 INFO:tasks.workunit.client.1.smithi169.stdout:2/86: fsync d0/d5/f13 0 2024-09-10T03:42:31.398 INFO:tasks.workunit.client.1.smithi169.stdout:7/82: dwrite d0/dd/f12 [0,4194304] 0 2024-09-10T03:42:31.398 INFO:tasks.workunit.client.1.smithi169.stdout:4/88: dwrite d0/d2/f9 [0,4194304] 0 2024-09-10T03:42:31.400 INFO:tasks.workunit.client.1.smithi169.stdout:3/106: mknod d0/c1f 0 2024-09-10T03:42:31.421 INFO:tasks.workunit.client.1.smithi169.stdout:3/107: dwrite d0/f7 [0,4194304] 0 2024-09-10T03:42:31.421 INFO:tasks.workunit.client.1.smithi169.stdout:3/108: chown d0/dc/d17 250705 1 2024-09-10T03:42:31.421 INFO:tasks.workunit.client.1.smithi169.stdout:3/109: fdatasync d0/f1a 0 2024-09-10T03:42:31.421 INFO:tasks.workunit.client.1.smithi169.stdout:3/110: fdatasync d0/dc/f14 0 2024-09-10T03:42:31.425 INFO:tasks.workunit.client.1.smithi169.stdout:3/111: symlink d0/dc/d17/l20 0 2024-09-10T03:42:31.427 INFO:tasks.workunit.client.1.smithi169.stdout:3/112: creat d0/dc/d17/f21 x:0 0 0 2024-09-10T03:42:31.430 INFO:tasks.workunit.client.1.smithi169.stdout:8/95: truncate f2 3258192 0 2024-09-10T03:42:31.431 INFO:tasks.workunit.client.1.smithi169.stdout:3/113: dread d0/fa [0,4194304] 0 2024-09-10T03:42:31.432 INFO:tasks.workunit.client.1.smithi169.stdout:8/96: symlink d3/de/l18 0 2024-09-10T03:42:31.436 INFO:tasks.workunit.client.1.smithi169.stdout:0/155: rename d5/db/d2d to d5/db/d20/d3f 0 2024-09-10T03:42:31.437 INFO:tasks.workunit.client.1.smithi169.stdout:0/156: dread d5/db/f25 [0,4194304] 0 2024-09-10T03:42:31.439 INFO:tasks.workunit.client.1.smithi169.stdout:7/83: rename d0/lf to d0/dd/l18 0 2024-09-10T03:42:31.440 INFO:tasks.workunit.client.1.smithi169.stdout:7/84: read - d0/dd/f15 zero size 2024-09-10T03:42:31.440 INFO:tasks.workunit.client.1.smithi169.stdout:7/85: chown d0/f3 589 1 2024-09-10T03:42:31.441 INFO:tasks.workunit.client.1.smithi169.stdout:7/86: creat d0/d10/f19 x:0 0 0 2024-09-10T03:42:31.450 INFO:tasks.workunit.client.1.smithi169.stdout:3/114: dwrite d0/f1a [0,4194304] 0 2024-09-10T03:42:31.450 INFO:tasks.workunit.client.1.smithi169.stdout:3/115: write d0/dc/f16 [563856,92741] 0 2024-09-10T03:42:31.455 INFO:tasks.workunit.client.1.smithi169.stdout:3/116: rename d0/fa to d0/dc/d17/f22 0 2024-09-10T03:42:31.460 INFO:tasks.workunit.client.1.smithi169.stdout:3/117: stat d0/c9 0 2024-09-10T03:42:31.460 INFO:tasks.workunit.client.1.smithi169.stdout:5/72: dread d3/f12 [0,4194304] 0 2024-09-10T03:42:31.461 INFO:tasks.workunit.client.1.smithi169.stdout:3/118: unlink d0/f7 0 2024-09-10T03:42:31.461 INFO:tasks.workunit.client.1.smithi169.stdout:3/119: write d0/dc/d17/f1e [341016,72958] 0 2024-09-10T03:42:31.462 INFO:tasks.workunit.client.1.smithi169.stdout:7/87: dwrite d0/fa [4194304,4194304] 0 2024-09-10T03:42:31.462 INFO:tasks.workunit.client.1.smithi169.stdout:0/157: dwrite d5/df/d13/f1d [0,4194304] 0 2024-09-10T03:42:31.462 INFO:tasks.workunit.client.1.smithi169.stdout:5/73: symlink d3/da/df/d14/l16 0 2024-09-10T03:42:31.471 INFO:tasks.workunit.client.1.smithi169.stdout:0/158: link l2 d5/db/l40 0 2024-09-10T03:42:31.472 INFO:tasks.workunit.client.1.smithi169.stdout:3/120: dread d0/dc/f15 [0,4194304] 0 2024-09-10T03:42:31.473 INFO:tasks.workunit.client.1.smithi169.stdout:5/74: mknod d3/da/c17 0 2024-09-10T03:42:31.476 INFO:tasks.workunit.client.1.smithi169.stdout:3/121: creat d0/f23 x:0 0 0 2024-09-10T03:42:31.477 INFO:tasks.workunit.client.1.smithi169.stdout:5/75: mkdir d3/da/df/d14/d18 0 2024-09-10T03:42:31.480 INFO:tasks.workunit.client.1.smithi169.stdout:3/122: mkdir d0/d1d/d24 0 2024-09-10T03:42:31.480 INFO:tasks.workunit.client.1.smithi169.stdout:3/123: dread - d0/f23 zero size 2024-09-10T03:42:31.481 INFO:tasks.workunit.client.1.smithi169.stdout:5/76: creat d3/da/df/d13/f19 x:0 0 0 2024-09-10T03:42:31.484 INFO:tasks.workunit.client.1.smithi169.stdout:5/77: dread d3/da/fd [0,4194304] 0 2024-09-10T03:42:31.484 INFO:tasks.workunit.client.1.smithi169.stdout:5/78: chown d3/c4 4107222 1 2024-09-10T03:42:31.488 INFO:tasks.workunit.client.1.smithi169.stdout:5/79: dread d3/da/fd [0,4194304] 0 2024-09-10T03:42:31.488 INFO:tasks.workunit.client.1.smithi169.stdout:5/80: chown d3/fc 326250 1 2024-09-10T03:42:31.488 INFO:tasks.workunit.client.1.smithi169.stdout:5/81: read - d3/da/df/d13/f19 zero size 2024-09-10T03:42:31.489 INFO:tasks.workunit.client.1.smithi169.stdout:0/159: dwrite d5/f9 [4194304,4194304] 0 2024-09-10T03:42:31.489 INFO:tasks.workunit.client.1.smithi169.stdout:7/88: dwrite d0/dd/f15 [0,4194304] 0 2024-09-10T03:42:31.489 INFO:tasks.workunit.client.1.smithi169.stdout:7/89: readlink d0/d10/l14 0 2024-09-10T03:42:31.490 INFO:tasks.workunit.client.1.smithi169.stdout:7/90: read d0/f2 [1003808,124407] 0 2024-09-10T03:42:31.493 INFO:tasks.workunit.client.1.smithi169.stdout:0/160: unlink d5/df/d2c/f3c 0 2024-09-10T03:42:31.496 INFO:tasks.workunit.client.1.smithi169.stdout:6/89: sync 2024-09-10T03:42:31.496 INFO:tasks.workunit.client.1.smithi169.stdout:1/79: sync 2024-09-10T03:42:31.497 INFO:tasks.workunit.client.1.smithi169.stdout:3/124: dwrite d0/dc/fe [0,4194304] 0 2024-09-10T03:42:31.506 INFO:tasks.workunit.client.1.smithi169.stdout:0/161: creat d5/db/d20/f41 x:0 0 0 2024-09-10T03:42:31.508 INFO:tasks.workunit.client.1.smithi169.stdout:1/80: truncate fa 782666 0 2024-09-10T03:42:31.510 INFO:tasks.workunit.client.1.smithi169.stdout:0/162: creat d5/df/d1b/d33/f42 x:0 0 0 2024-09-10T03:42:31.510 INFO:tasks.workunit.client.1.smithi169.stdout:6/90: mknod d0/d10/c1e 0 2024-09-10T03:42:31.510 INFO:tasks.workunit.client.1.smithi169.stdout:6/91: write d0/fc [4532890,19689] 0 2024-09-10T03:42:31.511 INFO:tasks.workunit.client.1.smithi169.stdout:6/92: dread - d0/d12/f1c zero size 2024-09-10T03:42:31.512 INFO:tasks.workunit.client.1.smithi169.stdout:0/163: mknod d5/db/c43 0 2024-09-10T03:42:31.514 INFO:tasks.workunit.client.1.smithi169.stdout:6/93: dread d0/d6/f13 [0,4194304] 0 2024-09-10T03:42:31.514 INFO:tasks.workunit.client.1.smithi169.stdout:6/94: chown d0/fc 3623155 1 2024-09-10T03:42:31.516 INFO:tasks.workunit.client.1.smithi169.stdout:7/91: dwrite d0/f5 [0,4194304] 0 2024-09-10T03:42:31.516 INFO:tasks.workunit.client.1.smithi169.stdout:7/92: stat d0/dd/f15 0 2024-09-10T03:42:31.529 INFO:tasks.workunit.client.1.smithi169.stdout:7/93: dread d0/dd/f12 [0,4194304] 0 2024-09-10T03:42:31.529 INFO:tasks.workunit.client.1.smithi169.stdout:7/94: dread - d0/d10/f19 zero size 2024-09-10T03:42:31.529 INFO:tasks.workunit.client.1.smithi169.stdout:7/95: stat d0/l4 0 2024-09-10T03:42:31.529 INFO:tasks.workunit.client.1.smithi169.stdout:7/96: chown d0/l7 58320 1 2024-09-10T03:42:31.529 INFO:tasks.workunit.client.1.smithi169.stdout:7/97: readlink d0/l7 0 2024-09-10T03:42:31.529 INFO:tasks.workunit.client.1.smithi169.stdout:7/98: chown d0/dd/f15 59 1 2024-09-10T03:42:31.529 INFO:tasks.workunit.client.1.smithi169.stdout:7/99: stat d0/dd/l11 0 2024-09-10T03:42:31.529 INFO:tasks.workunit.client.1.smithi169.stdout:7/100: dread - d0/d10/f19 zero size 2024-09-10T03:42:31.529 INFO:tasks.workunit.client.1.smithi169.stdout:7/101: chown d0/dd/f15 11607 1 2024-09-10T03:42:31.530 INFO:tasks.workunit.client.1.smithi169.stdout:7/102: creat d0/d10/f1a x:0 0 0 2024-09-10T03:42:31.530 INFO:tasks.workunit.client.1.smithi169.stdout:7/103: truncate d0/f3 1871907 0 2024-09-10T03:42:31.532 INFO:tasks.workunit.client.1.smithi169.stdout:7/104: link d0/l13 d0/l1b 0 2024-09-10T03:42:31.539 INFO:tasks.workunit.client.1.smithi169.stdout:0/164: dwrite d5/df/f26 [0,4194304] 0 2024-09-10T03:42:31.541 INFO:tasks.workunit.client.1.smithi169.stdout:1/81: dwrite fb [0,4194304] 0 2024-09-10T03:42:31.545 INFO:tasks.workunit.client.1.smithi169.stdout:0/165: creat d5/db/d24/f44 x:0 0 0 2024-09-10T03:42:31.547 INFO:tasks.workunit.client.1.smithi169.stdout:9/89: sync 2024-09-10T03:42:31.553 INFO:tasks.workunit.client.1.smithi169.stdout:1/82: mknod d12/c17 0 2024-09-10T03:42:31.556 INFO:tasks.workunit.client.1.smithi169.stdout:1/83: creat d12/f18 x:0 0 0 2024-09-10T03:42:31.557 INFO:tasks.workunit.client.1.smithi169.stdout:7/105: dwrite d0/dd/f15 [0,4194304] 0 2024-09-10T03:42:31.558 INFO:tasks.workunit.client.1.smithi169.stdout:8/97: dwrite f2 [0,4194304] 0 2024-09-10T03:42:31.558 INFO:tasks.workunit.client.1.smithi169.stdout:8/98: write d3/de/f16 [984384,102242] 0 2024-09-10T03:42:31.560 INFO:tasks.workunit.client.1.smithi169.stdout:7/106: rename d0/d10/l14 to d0/d16/l1c 0 2024-09-10T03:42:31.579 INFO:tasks.workunit.client.1.smithi169.stdout:6/95: dwrite d0/fc [0,4194304] 0 2024-09-10T03:42:31.585 INFO:tasks.workunit.client.1.smithi169.stdout:9/90: dwrite db/d10/f11 [0,4194304] 0 2024-09-10T03:42:31.586 INFO:tasks.workunit.client.1.smithi169.stdout:0/166: dwrite d5/db/d24/f44 [0,4194304] 0 2024-09-10T03:42:31.586 INFO:tasks.workunit.client.1.smithi169.stdout:0/167: stat d5/db/d24 0 2024-09-10T03:42:31.590 INFO:tasks.workunit.client.1.smithi169.stdout:0/168: dread d5/db/d20/d3f/f30 [0,4194304] 0 2024-09-10T03:42:31.593 INFO:tasks.workunit.client.1.smithi169.stdout:9/91: dread db/fc [0,4194304] 0 2024-09-10T03:42:31.594 INFO:tasks.workunit.client.1.smithi169.stdout:7/107: dwrite d0/f5 [0,4194304] 0 2024-09-10T03:42:31.595 INFO:tasks.workunit.client.1.smithi169.stdout:6/96: dwrite d0/d10/f19 [0,4194304] 0 2024-09-10T03:42:31.595 INFO:tasks.workunit.client.1.smithi169.stdout:6/97: dread - d0/d10/f1d zero size 2024-09-10T03:42:31.598 INFO:tasks.workunit.client.1.smithi169.stdout:2/87: sync 2024-09-10T03:42:31.599 INFO:tasks.workunit.client.1.smithi169.stdout:4/89: sync 2024-09-10T03:42:31.599 INFO:tasks.workunit.client.1.smithi169.stdout:0/169: getdents d5/df/d2c 0 2024-09-10T03:42:31.600 INFO:tasks.workunit.client.1.smithi169.stdout:7/108: mknod d0/d10/c1d 0 2024-09-10T03:42:31.605 INFO:tasks.workunit.client.1.smithi169.stdout:4/90: rmdir d0/d6 39 2024-09-10T03:42:31.605 INFO:tasks.workunit.client.1.smithi169.stdout:4/91: readlink d0/d6/l1b 0 2024-09-10T03:42:31.607 INFO:tasks.workunit.client.1.smithi169.stdout:2/88: link d0/d5/f13 d0/d16/f1a 0 2024-09-10T03:42:31.607 INFO:tasks.workunit.client.1.smithi169.stdout:2/89: chown d0/d5/l10 1941916 1 2024-09-10T03:42:31.607 INFO:tasks.workunit.client.1.smithi169.stdout:2/90: write d0/d5/f13 [112824,31330] 0 2024-09-10T03:42:31.608 INFO:tasks.workunit.client.1.smithi169.stdout:9/92: dwrite db/d10/f11 [0,4194304] 0 2024-09-10T03:42:31.610 INFO:tasks.workunit.client.1.smithi169.stdout:0/170: mkdir d5/df/d2c/d45 0 2024-09-10T03:42:31.610 INFO:tasks.workunit.client.1.smithi169.stdout:0/171: fsync d5/f7 0 2024-09-10T03:42:31.610 INFO:tasks.workunit.client.1.smithi169.stdout:0/172: write d5/db/f21 [145335,104938] 0 2024-09-10T03:42:31.610 INFO:tasks.workunit.client.1.smithi169.stdout:7/109: rmdir d0/dd 39 2024-09-10T03:42:31.610 INFO:tasks.workunit.client.1.smithi169.stdout:7/110: chown d0/dd/f15 10 1 2024-09-10T03:42:31.610 INFO:tasks.workunit.client.1.smithi169.stdout:7/111: read - d0/d10/f1a zero size 2024-09-10T03:42:31.613 INFO:tasks.workunit.client.1.smithi169.stdout:6/98: mkdir d0/d10/d1f 0 2024-09-10T03:42:31.613 INFO:tasks.workunit.client.1.smithi169.stdout:6/99: write d0/d10/f19 [4202611,34889] 0 2024-09-10T03:42:31.613 INFO:tasks.workunit.client.1.smithi169.stdout:6/100: truncate d0/d10/f1d 622490 0 2024-09-10T03:42:31.614 INFO:tasks.workunit.client.1.smithi169.stdout:9/93: creat db/f1c x:0 0 0 2024-09-10T03:42:31.614 INFO:tasks.workunit.client.1.smithi169.stdout:9/94: write db/f1c [815083,23060] 0 2024-09-10T03:42:31.616 INFO:tasks.workunit.client.1.smithi169.stdout:0/173: creat d5/db/d20/d3f/d3a/f46 x:0 0 0 2024-09-10T03:42:31.616 INFO:tasks.workunit.client.1.smithi169.stdout:7/112: mkdir d0/d16/d1e 0 2024-09-10T03:42:31.622 INFO:tasks.workunit.client.1.smithi169.stdout:2/91: read d0/f15 [4113805,26812] 0 2024-09-10T03:42:31.622 INFO:tasks.workunit.client.1.smithi169.stdout:0/174: creat d5/db/d20/d3f/d3a/f47 x:0 0 0 2024-09-10T03:42:31.622 INFO:tasks.workunit.client.1.smithi169.stdout:6/101: rename d0/d10/f19 to d0/d10/f20 0 2024-09-10T03:42:31.625 INFO:tasks.workunit.client.1.smithi169.stdout:7/113: write d0/fa [6999829,55816] 0 2024-09-10T03:42:31.625 INFO:tasks.workunit.client.1.smithi169.stdout:2/92: truncate d0/d5/fa 808388 0 2024-09-10T03:42:31.626 INFO:tasks.workunit.client.1.smithi169.stdout:2/93: chown d0/f2 528 1 2024-09-10T03:42:31.627 INFO:tasks.workunit.client.1.smithi169.stdout:0/175: symlink d5/db/d20/l48 0 2024-09-10T03:42:31.627 INFO:tasks.workunit.client.1.smithi169.stdout:0/176: readlink d5/db/d20/l2a 0 2024-09-10T03:42:31.628 INFO:tasks.workunit.client.1.smithi169.stdout:6/102: creat d0/d15/f21 x:0 0 0 2024-09-10T03:42:31.630 INFO:tasks.workunit.client.1.smithi169.stdout:7/114: link d0/dd/l18 d0/d10/l1f 0 2024-09-10T03:42:31.631 INFO:tasks.workunit.client.1.smithi169.stdout:0/177: rmdir d5/db/d20/d3f 39 2024-09-10T03:42:31.633 INFO:tasks.workunit.client.1.smithi169.stdout:6/103: dread d0/f9 [4194304,4194304] 0 2024-09-10T03:42:31.633 INFO:tasks.workunit.client.1.smithi169.stdout:0/178: creat d5/df/d2c/f49 x:0 0 0 2024-09-10T03:42:31.633 INFO:tasks.workunit.client.1.smithi169.stdout:0/179: read - d5/db/d20/d3f/d3a/f46 zero size 2024-09-10T03:42:31.634 INFO:tasks.workunit.client.1.smithi169.stdout:4/92: dwrite d0/d2/f8 [0,4194304] 0 2024-09-10T03:42:31.636 INFO:tasks.workunit.client.1.smithi169.stdout:6/104: rename d0/d6/f13 to d0/d6/f22 0 2024-09-10T03:42:31.637 INFO:tasks.workunit.client.1.smithi169.stdout:2/94: dread d0/d5/fe [0,4194304] 0 2024-09-10T03:42:31.637 INFO:tasks.workunit.client.1.smithi169.stdout:2/95: chown d0/d5/fe 63 1 2024-09-10T03:42:31.640 INFO:tasks.workunit.client.1.smithi169.stdout:7/115: write d0/fa [3136271,35339] 0 2024-09-10T03:42:31.657 INFO:tasks.workunit.client.1.smithi169.stdout:6/105: rmdir d0/d15 39 2024-09-10T03:42:31.657 INFO:tasks.workunit.client.1.smithi169.stdout:2/96: dwrite d0/f18 [0,4194304] 0 2024-09-10T03:42:31.662 INFO:tasks.workunit.client.1.smithi169.stdout:6/106: creat d0/d10/d1f/f23 x:0 0 0 2024-09-10T03:42:31.665 INFO:tasks.workunit.client.1.smithi169.stdout:2/97: read d0/d5/ff [932919,71875] 0 2024-09-10T03:42:31.665 INFO:tasks.workunit.client.1.smithi169.stdout:2/98: chown d0 432004687 1 2024-09-10T03:42:31.670 INFO:tasks.workunit.client.1.smithi169.stdout:7/116: truncate d0/dd/f15 400163 0 2024-09-10T03:42:31.671 INFO:tasks.workunit.client.1.smithi169.stdout:7/117: chown d0/dd/l18 20 1 2024-09-10T03:42:31.674 INFO:tasks.workunit.client.1.smithi169.stdout:6/107: mkdir d0/d6/d18/d24 0 2024-09-10T03:42:31.676 INFO:tasks.workunit.client.1.smithi169.stdout:2/99: rename d0/d16/f1a to d0/d16/d19/f1b 0 2024-09-10T03:42:31.676 INFO:tasks.workunit.client.1.smithi169.stdout:6/108: rename d0/d12 to d0/d12/d25 22 2024-09-10T03:42:31.676 INFO:tasks.workunit.client.1.smithi169.stdout:2/100: chown d0/d5/l10 0 1 2024-09-10T03:42:31.679 INFO:tasks.workunit.client.1.smithi169.stdout:7/118: mkdir d0/d16/d20 0 2024-09-10T03:42:31.682 INFO:tasks.workunit.client.1.smithi169.stdout:2/101: rmdir d0/d16/d19 39 2024-09-10T03:42:31.683 INFO:tasks.workunit.client.1.smithi169.stdout:6/109: mknod d0/d6/d18/c26 0 2024-09-10T03:42:31.683 INFO:tasks.workunit.client.1.smithi169.stdout:6/110: dread - d0/d15/f21 zero size 2024-09-10T03:42:31.686 INFO:tasks.workunit.client.1.smithi169.stdout:7/119: rename d0/d10/d17 to d0/dd/d21 0 2024-09-10T03:42:31.686 INFO:tasks.workunit.client.1.smithi169.stdout:2/102: creat d0/d16/d19/f1c x:0 0 0 2024-09-10T03:42:31.688 INFO:tasks.workunit.client.1.smithi169.stdout:6/111: creat d0/d10/d1f/f27 x:0 0 0 2024-09-10T03:42:31.690 INFO:tasks.workunit.client.1.smithi169.stdout:6/112: dread d0/d6/f22 [0,4194304] 0 2024-09-10T03:42:31.691 INFO:tasks.workunit.client.1.smithi169.stdout:2/103: rename d0/d17 to d0/d1d 0 2024-09-10T03:42:31.692 INFO:tasks.workunit.client.1.smithi169.stdout:2/104: write d0/d5/ff [1475636,96422] 0 2024-09-10T03:42:31.694 INFO:tasks.workunit.client.1.smithi169.stdout:6/113: mkdir d0/d6/d28 0 2024-09-10T03:42:31.694 INFO:tasks.workunit.client.1.smithi169.stdout:6/114: readlink d0/l2 0 2024-09-10T03:42:31.697 INFO:tasks.workunit.client.1.smithi169.stdout:2/105: mkdir d0/d1e 0 2024-09-10T03:42:31.699 INFO:tasks.workunit.client.1.smithi169.stdout:6/115: symlink d0/d6/d18/d24/l29 0 2024-09-10T03:42:31.701 INFO:tasks.workunit.client.1.smithi169.stdout:2/106: mknod d0/d16/c1f 0 2024-09-10T03:42:31.703 INFO:tasks.workunit.client.1.smithi169.stdout:0/180: write d5/db/d24/f44 [4162286,70723] 0 2024-09-10T03:42:31.707 INFO:tasks.workunit.client.1.smithi169.stdout:0/181: truncate d5/db/d20/d3f/f36 92498 0 2024-09-10T03:42:31.711 INFO:tasks.workunit.client.1.smithi169.stdout:0/182: creat d5/df/d1b/d33/f4a x:0 0 0 2024-09-10T03:42:31.714 INFO:tasks.workunit.client.1.smithi169.stdout:0/183: rmdir d5/db/d20 39 2024-09-10T03:42:31.714 INFO:tasks.workunit.client.1.smithi169.stdout:0/184: dread - d5/df/d1b/d33/f42 zero size 2024-09-10T03:42:31.714 INFO:tasks.workunit.client.1.smithi169.stdout:0/185: truncate d5/db/f2b 704688 0 2024-09-10T03:42:31.717 INFO:tasks.workunit.client.1.smithi169.stdout:0/186: creat d5/df/d13/f4b x:0 0 0 2024-09-10T03:42:31.717 INFO:tasks.workunit.client.1.smithi169.stdout:0/187: chown d5/df/d1b/d33 42762 1 2024-09-10T03:42:31.721 INFO:tasks.workunit.client.1.smithi169.stdout:0/188: symlink d5/df/d1b/d33/l4c 0 2024-09-10T03:42:31.721 INFO:tasks.workunit.client.1.smithi169.stdout:0/189: stat d5/db/f2b 0 2024-09-10T03:42:31.722 INFO:tasks.workunit.client.1.smithi169.stdout:0/190: dread d5/db/f25 [0,4194304] 0 2024-09-10T03:42:31.724 INFO:tasks.workunit.client.1.smithi169.stdout:6/116: dwrite d0/d10/d1f/f27 [0,4194304] 0 2024-09-10T03:42:31.725 INFO:tasks.workunit.client.1.smithi169.stdout:6/117: dread - d0/d12/f1c zero size 2024-09-10T03:42:31.725 INFO:tasks.workunit.client.1.smithi169.stdout:0/191: mknod d5/df/c4d 0 2024-09-10T03:42:31.725 INFO:tasks.workunit.client.1.smithi169.stdout:0/192: truncate d5/db/f25 914886 0 2024-09-10T03:42:31.728 INFO:tasks.workunit.client.1.smithi169.stdout:9/95: truncate db/d10/f11 3923317 0 2024-09-10T03:42:31.730 INFO:tasks.workunit.client.1.smithi169.stdout:2/107: write d0/d5/fa [541636,40058] 0 2024-09-10T03:42:31.735 INFO:tasks.workunit.client.1.smithi169.stdout:9/96: creat db/d10/f1d x:0 0 0 2024-09-10T03:42:31.735 INFO:tasks.workunit.client.1.smithi169.stdout:9/97: fsync db/f19 0 2024-09-10T03:42:31.735 INFO:tasks.workunit.client.1.smithi169.stdout:9/98: write db/fc [2789334,128591] 0 2024-09-10T03:42:31.735 INFO:tasks.workunit.client.1.smithi169.stdout:2/108: symlink d0/l20 0 2024-09-10T03:42:31.735 INFO:tasks.workunit.client.1.smithi169.stdout:2/109: dread - d0/d16/d19/f1c zero size 2024-09-10T03:42:31.736 INFO:tasks.workunit.client.1.smithi169.stdout:2/110: write d0/f11 [217616,31896] 0 2024-09-10T03:42:31.739 INFO:tasks.workunit.client.1.smithi169.stdout:9/99: symlink db/d10/l1e 0 2024-09-10T03:42:31.740 INFO:tasks.workunit.client.1.smithi169.stdout:0/193: write d5/df/d13/f1d [2977312,78345] 0 2024-09-10T03:42:31.741 INFO:tasks.workunit.client.1.smithi169.stdout:2/111: symlink d0/l21 0 2024-09-10T03:42:31.741 INFO:tasks.workunit.client.1.smithi169.stdout:2/112: chown d0/d5/c9 1 1 2024-09-10T03:42:31.741 INFO:tasks.workunit.client.1.smithi169.stdout:2/113: chown d0/l20 2 1 2024-09-10T03:42:31.742 INFO:tasks.workunit.client.1.smithi169.stdout:9/100: truncate db/ff 131026 0 2024-09-10T03:42:31.743 INFO:tasks.workunit.client.1.smithi169.stdout:6/118: dwrite d0/d10/f1d [0,4194304] 0 2024-09-10T03:42:31.746 INFO:tasks.workunit.client.1.smithi169.stdout:0/194: creat d5/df/f4e x:0 0 0 2024-09-10T03:42:31.746 INFO:tasks.workunit.client.1.smithi169.stdout:0/195: chown d5/db/d20/d3f/d3a 0 1 2024-09-10T03:42:31.748 INFO:tasks.workunit.client.1.smithi169.stdout:9/101: rename db/d10/c14 to db/c1f 0 2024-09-10T03:42:31.749 INFO:tasks.workunit.client.1.smithi169.stdout:9/102: read db/f19 [332031,113954] 0 2024-09-10T03:42:31.749 INFO:tasks.workunit.client.1.smithi169.stdout:9/103: dread - db/d10/f1d zero size 2024-09-10T03:42:31.753 INFO:tasks.workunit.client.1.smithi169.stdout:0/196: mkdir d5/d4f 0 2024-09-10T03:42:31.755 INFO:tasks.workunit.client.1.smithi169.stdout:0/197: symlink d5/df/d13/l50 0 2024-09-10T03:42:31.755 INFO:tasks.workunit.client.1.smithi169.stdout:0/198: fdatasync d5/df/d13/f19 0 2024-09-10T03:42:31.776 INFO:tasks.workunit.client.1.smithi169.stdout:6/119: dwrite d0/d6/f1b [0,4194304] 0 2024-09-10T03:42:31.776 INFO:tasks.workunit.client.1.smithi169.stdout:6/120: chown d0/d6/d18 211 1 2024-09-10T03:42:31.776 INFO:tasks.workunit.client.1.smithi169.stdout:2/114: dwrite d0/f11 [0,4194304] 0 2024-09-10T03:42:31.776 INFO:tasks.workunit.client.1.smithi169.stdout:2/115: chown d0/c14 241 1 2024-09-10T03:42:31.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:31 smithi169 ceph-mon[35668]: Deploying cephadm binary to smithi112 2024-09-10T03:42:31.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:31 smithi169 ceph-mon[35668]: Deploying cephadm binary to smithi169 2024-09-10T03:42:31.776 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:31 smithi169 ceph-mon[35668]: [10/Sep/2024:03:42:30] ENGINE Bus STARTING 2024-09-10T03:42:31.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:31 smithi169 ceph-mon[35668]: pgmap v4: 65 pgs: 65 active+clean; 206 MiB data, 894 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:42:31.777 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Sep 10 03:42:31 smithi169 ceph-mon[35668]: mgrmap e23: smithi169.bkwgjl(active, since 2s) 2024-09-10T03:42:31.777 INFO:tasks.workunit.client.1.smithi169.stdout:9/104: dwrite db/fc [0,4194304] 0 2024-09-10T03:42:31.777 INFO:tasks.workunit.client.1.smithi169.stdout:9/105: chown db/ld 0 1 2024-09-10T03:42:31.778 INFO:tasks.workunit.client.1.smithi169.stdout:6/121: dread d0/d6/f22 [0,4194304] 0 2024-09-10T03:42:31.780 INFO:tasks.workunit.client.1.smithi169.stdout:2/116: creat d0/d1d/f22 x:0 0 0 2024-09-10T03:42:31.780 INFO:tasks.workunit.client.1.smithi169.stdout:7/120: dread d0/dd/f15 [0,4194304] 0 2024-09-10T03:42:31.780 INFO:tasks.workunit.client.1.smithi169.stdout:7/121: fsync d0/f5 0 2024-09-10T03:42:31.783 INFO:tasks.workunit.client.1.smithi169.stdout:7/122: unlink d0/d10/l1f 0 2024-09-10T03:42:31.784 INFO:tasks.workunit.client.1.smithi169.stdout:6/122: rename d0/d10/d1f to d0/d6/d18/d24/d2a 0 2024-09-10T03:42:31.785 INFO:tasks.workunit.client.1.smithi169.stdout:7/123: creat d0/dd/f22 x:0 0 0 2024-09-10T03:42:31.787 INFO:tasks.workunit.client.1.smithi169.stdout:2/117: rename d0/d16/d19/f1b to d0/d1e/f23 0 2024-09-10T03:42:31.787 INFO:tasks.workunit.client.1.smithi169.stdout:2/118: chown d0/f11 81990 1 2024-09-10T03:42:31.792 INFO:tasks.workunit.client.1.smithi169.stdout:2/119: creat d0/d1e/f24 x:0 0 0 2024-09-10T03:42:31.792 INFO:tasks.workunit.client.1.smithi169.stdout:2/120: truncate d0/d16/d19/f1c 495642 0 2024-09-10T03:42:31.793 INFO:tasks.workunit.client.1.smithi169.stdout:9/106: dwrite f7 [8388608,4194304] 0 2024-09-10T03:42:31.795 INFO:tasks.workunit.client.1.smithi169.stdout:2/121: mknod d0/d16/c25 0 2024-09-10T03:42:31.800 INFO:tasks.workunit.client.1.smithi169.stdout:2/122: creat d0/d5/f26 x:0 0 0 2024-09-10T03:42:31.800 INFO:tasks.workunit.client.1.smithi169.stdout:2/123: fdatasync d0/fd 0 2024-09-10T03:42:31.806 INFO:tasks.workunit.client.1.smithi169.stdout:9/107: dwrite db/fe [0,4194304] 0 2024-09-10T03:42:31.820 INFO:tasks.workunit.client.1.smithi169.stdout:0/199: dwrite d5/db/d20/d3f/f36 [0,4194304] 0 2024-09-10T03:42:31.820 INFO:tasks.workunit.client.1.smithi169.stdout:0/200: chown d5/db/d20/d3f 7525 1 2024-09-10T03:42:31.820 INFO:tasks.workunit.client.1.smithi169.stdout:8/99: sync 2024-09-10T03:42:31.820 INFO:tasks.workunit.client.1.smithi169.stdout:3/125: sync 2024-09-10T03:42:31.821 INFO:tasks.workunit.client.1.smithi169.stdout:5/82: sync 2024-09-10T03:42:31.821 INFO:tasks.workunit.client.1.smithi169.stdout:1/84: sync 2024-09-10T03:42:31.821 INFO:tasks.workunit.client.1.smithi169.stdout:1/85: fdatasync f8 0 2024-09-10T03:42:31.821 INFO:tasks.workunit.client.1.smithi169.stdout:4/93: sync 2024-09-10T03:42:31.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:31 smithi112 ceph-mon[27737]: Deploying cephadm binary to smithi112 2024-09-10T03:42:31.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:31 smithi112 ceph-mon[27737]: Deploying cephadm binary to smithi169 2024-09-10T03:42:31.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:31 smithi112 ceph-mon[27737]: [10/Sep/2024:03:42:30] ENGINE Bus STARTING 2024-09-10T03:42:31.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:31 smithi112 ceph-mon[27737]: pgmap v4: 65 pgs: 65 active+clean; 206 MiB data, 894 MiB used, 536 GiB / 536 GiB avail 2024-09-10T03:42:31.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:42:31 smithi112 ceph-mon[27737]: mgrmap e23: smithi169.bkwgjl(active, since 2s) 2024-09-10T03:42:31.832 INFO:tasks.workunit.client.1.smithi169.stdout:0/201: rmdir d5/df/d2c/d45 0 2024-09-10T03:42:31.834 INFO:tasks.workunit.client.1.smithi169.stdout:3/126: mknod d0/d1d/c25 0 2024-09-10T03:42:31.834 INFO:tasks.workunit.client.1.smithi169.stdout:1/86: dwrite f3 [4194304,4194304] 0 2024-09-10T03:42:31.835 INFO:tasks.workunit.client.1.smithi169.stdout:8/100: mkdir d3/d19 0 2024-09-10T03:42:31.835 INFO:tasks.workunit.client.1.smithi169.stdout:8/101: stat d3/ca 0 2024-09-10T03:42:31.837 INFO:tasks.workunit.client.1.smithi169.stdout:5/83: creat d3/da/f1a x:0 0 0 2024-09-10T03:42:31.838 INFO:tasks.workunit.client.1.smithi169.stdout:4/94: symlink d0/d1d/l21 0 2024-09-10T03:42:31.844 INFO:tasks.workunit.client.1.smithi169.stdout:3/127: stat d0/dc/cf 0 2024-09-10T03:42:31.844 INFO:tasks.workunit.client.1.smithi169.stdout:3/128: write d0/f23 [1012797,31019] 0 2024-09-10T03:42:31.844 INFO:tasks.workunit.client.1.smithi169.stdout:3/129: dread - d0/dc/d17/f21 zero size 2024-09-10T03:42:31.844 INFO:tasks.workunit.client.1.smithi169.stdout:6/123: truncate d0/d10/f1d 3762197 0 2024-09-10T03:42:31.845 INFO:tasks.workunit.client.1.smithi169.stdout:0/202: getdents d5 0 2024-09-10T03:42:31.848 INFO:tasks.workunit.client.1.smithi169.stdout:1/87: rename d12/f16 to d12/f19 0 2024-09-10T03:42:31.849 INFO:tasks.workunit.client.1.smithi169.stdout:5/84: rmdir d3/da/df/d14 39 2024-09-10T03:42:31.849 INFO:tasks.workunit.client.1.smithi169.stdout:5/85: chown d3/fc 499898 1 2024-09-10T03:42:31.849 INFO:tasks.workunit.client.1.smithi169.stdout:4/95: dread d0/d2/f9 [0,4194304] 0 2024-09-10T03:42:31.851 INFO:tasks.workunit.client.1.smithi169.stdout:8/102: dwrite d3/de/f14 [0,4194304] 0 2024-09-10T03:42:31.851 INFO:tasks.workunit.client.1.smithi169.stdout:8/103: chown d3/de 442213 1 2024-09-10T03:42:31.851 INFO:tasks.workunit.client.1.smithi169.stdout:8/104: stat d3/dc 0 2024-09-10T03:42:31.856 INFO:tasks.workunit.client.1.smithi169.stdout:0/203: rename d5/c12 to d5/df/d1b/d33/c51 0 2024-09-10T03:42:31.858 INFO:tasks.workunit.client.1.smithi169.stdout:5/86: symlink d3/da/l1b 0 2024-09-10T03:42:31.858 INFO:tasks.workunit.client.1.smithi169.stdout:5/87: chown d3/da/c17 37 1 2024-09-10T03:42:31.858 INFO:tasks.workunit.client.1.smithi169.stdout:1/88: dread f3 [4194304,4194304] 0 2024-09-10T03:42:31.858 INFO:tasks.workunit.client.1.smithi169.stdout:1/89: read - d12/f18 zero size 2024-09-10T03:42:31.860 INFO:tasks.workunit.client.1.smithi169.stdout:4/96: creat d0/d16/f22 x:0 0 0 2024-09-10T03:42:31.862 INFO:tasks.workunit.client.1.smithi169.stdout:4/97: dread d0/d2/f14 [0,4194304] 0 2024-09-10T03:42:31.862 INFO:tasks.workunit.client.1.smithi169.stdout:3/130: link d0/dc/f16 d0/f26 0 2024-09-10T03:42:31.864 INFO:tasks.workunit.client.1.smithi169.stdout:3/131: dread d0/dc/f16 [0,4194304] 0 2024-09-10T03:42:31.867 INFO:tasks.workunit.client.1.smithi169.stdout:8/105: symlink d3/dc/l1a 0 2024-09-10T03:42:31.868 INFO:tasks.workunit.client.1.smithi169.stdout:0/204: read d5/db/f21 [31243,82220] 0 2024-09-10T03:42:31.872 INFO:tasks.workunit.client.1.smithi169.stdout:5/88: creat d3/da/df/d14/d18/f1c x:0 0 0 2024-09-10T03:42:31.872 INFO:tasks.workunit.client.1.smithi169.stdout:1/90: creat d12/f1a x:0 0 0 2024-09-10T03:42:31.873 INFO:tasks.workunit.client.1.smithi169.stdout:4/98: mkdir d0/d1d/d23 0 2024-09-10T03:42:34.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.836+0000 7f17f4b2d640 1 -- 172.21.15.112:0/1659532099 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17f0101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T03:42:34.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.836+0000 7f17f4b2d640 1 --2- 172.21.15.112:0/1659532099 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0101a40 0x7f17f0101ec0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f17e400ba90 tx=0x7f17e402c2c0 comp rx=0 tx=0).stop 2024-09-10T03:42:34.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.836+0000 7f17f4b2d640 1 -- 172.21.15.112:0/1659532099 shutdown_connections 2024-09-10T03:42:34.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.836+0000 7f17f4b2d640 1 --2- 172.21.15.112:0/1659532099 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0101a40 0x7f17f0101ec0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:34.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.836+0000 7f17f4b2d640 1 --2- 172.21.15.112:0/1659532099 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0100840 0x7f17f0100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:34.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.836+0000 7f17f4b2d640 1 -- 172.21.15.112:0/1659532099 >> 172.21.15.112:0/1659532099 conn(0x7f17f00fbff0 msgr2=0x7f17f00fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T03:42:34.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.837+0000 7f17f4b2d640 1 -- 172.21.15.112:0/1659532099 shutdown_connections 2024-09-10T03:42:34.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.837+0000 7f17f4b2d640 1 -- 172.21.15.112:0/1659532099 wait complete. 2024-09-10T03:42:34.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.838+0000 7f17f4b2d640 1 Processor -- start 2024-09-10T03:42:34.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.839+0000 7f17f4b2d640 1 -- start start 2024-09-10T03:42:34.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.839+0000 7f17f4b2d640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17f0195110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:34.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.839+0000 7f17f4b2d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17f0195650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:34.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.839+0000 7f17f4b2d640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17f01085f0 con 0x7f17f0100840 2024-09-10T03:42:34.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.839+0000 7f17f4b2d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0108470 con 0x7f17f0101a40 2024-09-10T03:42:34.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.839+0000 7f17ee59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17f0195110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:42:34.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.839+0000 7f17ee59c640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17f0195110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:39848/0 (socket says 172.21.15.112:39848) 2024-09-10T03:42:34.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.839+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 learned_addr learned my addr 172.21.15.112:0/2221466548 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-09-10T03:42:34.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.839+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17f0195650 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:42:34.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.839+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17f0195650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:34.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.839+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f0195cd0 con 0x7f17f0100840 2024-09-10T03:42:34.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.841+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17f0195110 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f17e000b3f0 tx=0x7f17e000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:42:34.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.841+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0015020 con 0x7f17f0100840 2024-09-10T03:42:34.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.841+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0011af0 con 0x7f17f0100840 2024-09-10T03:42:34.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.841+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0010cb0 con 0x7f17f0100840 2024-09-10T03:42:34.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.841+0000 7f17f4b2d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f019a710 con 0x7f17f0100840 2024-09-10T03:42:34.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.841+0000 7f17f4b2d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f019ad20 con 0x7f17f0100840 2024-09-10T03:42:34.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.843+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0018020 con 0x7f17f0100840 2024-09-10T03:42:34.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.843+0000 7f17f4b2d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17f00ce790 con 0x7f17f0100840 2024-09-10T03:42:34.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.843+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:34.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.843+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5812+0+0 (secure 0 0 0) 0x7f17e0054f40 con 0x7f17f0100840 2024-09-10T03:42:34.849 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:34.848+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195986 (secure 0 0 0) 0x7f17e001f020 con 0x7f17f0100840 2024-09-10T03:42:35.086 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:35.085+0000 7f17f4b2d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) -- 0x7f17f01064e0 con 0x7f17d803f9c0 2024-09-10T03:42:44.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:44.853+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.169:6828/1204620466, fault. 2024-09-10T03:42:44.854 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:44.853+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T03:42:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:52.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:52.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:52.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:53.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:53.100+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:53.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:53.100+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:53.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:53.100+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:54.068 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:53.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:54.068 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:53.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:54.069 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:53.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:54.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:54.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:54.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:55.066 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:54.716+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:55.066 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:54.716+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:55.066 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:54.716+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:55.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:55.064+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.169:6828/1204620466, fault. 2024-09-10T03:42:55.066 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:55.064+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T03:42:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:55.114+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:55.114+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:55.114+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:56.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:55.751+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:56.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:55.751+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:56.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:55.751+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:42:55.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:42:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:42:55.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:42:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:42:55.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:42:56.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:56.160+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:56.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:56.160+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:56.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:56.160+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:56.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.709+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17f0195110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:42:56.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.709+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17f0195110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:42:56.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.709+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17f0195110 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f17e000b3f0 tx=0x7f17e000b8b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:42:56.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.709+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17f0195110 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f17e000b3f0 tx=0x7f17e000b8b0 comp rx=0 tx=0).stop 2024-09-10T03:42:56.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.709+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17f0195110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:42:56.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.709+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17f0195110 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:56.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.709+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:56.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.709+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d804c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:42:56.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.709+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804ca40 con 0x7f17d8047320 2024-09-10T03:42:56.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.709+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804cbc0 con 0x7f17f0101a40 2024-09-10T03:42:56.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.709+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:42:56.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.709+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d804c030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:42:56.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.709+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d804c030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:42:56.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.709+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17e0010a80 con 0x7f17d8047320 2024-09-10T03:42:56.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.710+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f17e0053360 tx=0x7f17e0009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:42:56.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.710+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0011ca0 con 0x7f17d8047320 2024-09-10T03:42:56.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.711+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e000cd10 con 0x7f17d8047320 2024-09-10T03:42:56.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:42:56.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0021c00 con 0x7f17d8047320 2024-09-10T03:42:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:56.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:56.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:56.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:57.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:42:56.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:42:57.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:42:56.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:42:57.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:42:56.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:42:57.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:57.184+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:57.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:57.184+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:57.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:57.184+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:57.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:57.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:57.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:42:57.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:42:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:42:57.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:42:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:42:57.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:42:58.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:58.193+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:58.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:58.193+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:58.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:58.193+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:59.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:42:58.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:42:59.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:42:58.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:42:59.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:42:58.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:42:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:58.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:58.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:58.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:42:59.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:59.189+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:59.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:59.189+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:42:59.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:42:59.189+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:00.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:42:59.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:00.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:42:59.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:00.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:42:59.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:00.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:59.748+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:00.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:59.748+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:00.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:42:59.748+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:00.383 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:00.191+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:00.383 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:00.191+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:00.384 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:00.191+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:01.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:00.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:01.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:00.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:01.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:00.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:00.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:00.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:00.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:01.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:01.168+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:01.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:01.168+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:01.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:01.168+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:01.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.709+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T03:43:01.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.709+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:43:01.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.709+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f17e0053360 tx=0x7f17e0009190 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:43:01.710 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.709+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f17e0053360 tx=0x7f17e0009190 comp rx=0 tx=0).stop 2024-09-10T03:43:01.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.709+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:43:01.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.709+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:01.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.709+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:01.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.709+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d804c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:01.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.709+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804fbe0 con 0x7f17f0100840 2024-09-10T03:43:01.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.709+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804fd60 con 0x7f17f0101a40 2024-09-10T03:43:01.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.710+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:43:01.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.711+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d804c030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:43:01.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.711+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d804c030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:01.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.711+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17e0010a80 con 0x7f17f0100840 2024-09-10T03:43:01.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.711+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804f1a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f17e003dd40 tx=0x7f17e0054000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:43:01.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028020 con 0x7f17f0100840 2024-09-10T03:43:01.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0017c70 con 0x7f17f0100840 2024-09-10T03:43:01.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:01.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e002b020 con 0x7f17f0100840 2024-09-10T03:43:02.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:01.814+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:02.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:01.814+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:02.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:01.814+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:01.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:01.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:01.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:02.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:02.145+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:02.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:02.145+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:02.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:02.145+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:02.783+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:02.783+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:02.783+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:03.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:02.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:03.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:02.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:02.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:03.512 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:03.182+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:03.512 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:03.182+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:03.512 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:03.182+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:03.791 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:03.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:03.792 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:03.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:03.792 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:03.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:04.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:03.790+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:04.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:03.790+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:04.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:03.790+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:04.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:04.155+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:04.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:04.155+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:04.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:04.155+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:04.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:04.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:04.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:05.176 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:04.839+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:05.176 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:04.839+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:05.177 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:04.839+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:05.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:05.472+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.169:6828/1204620466, fault. 2024-09-10T03:43:05.473 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:05.472+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T03:43:05.473 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:05.175+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:05.473 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:05.175+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:05.473 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:05.175+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:05.792 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:05.559+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:05.792 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:05.559+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:05.792 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:05.559+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:06.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:05.791+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:06.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:05.791+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:06.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:05.791+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:06.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:06.159+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:06.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:06.159+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:06.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:06.159+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:06.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.709+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T03:43:06.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.709+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:43:06.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.709+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804f1a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f17e003dd40 tx=0x7f17e0054000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:43:06.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.709+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804f1a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f17e003dd40 tx=0x7f17e0054000 comp rx=0 tx=0).stop 2024-09-10T03:43:06.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.709+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804f1a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:43:06.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.709+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804f1a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:06.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.709+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d804c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:06.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.709+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:06.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.709+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8050280 con 0x7f17d8047320 2024-09-10T03:43:06.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.709+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8050400 con 0x7f17f0101a40 2024-09-10T03:43:06.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.710+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:43:06.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.710+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d804c030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:43:06.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.710+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d804c030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:06.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.710+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17e0010a80 con 0x7f17d8047320 2024-09-10T03:43:06.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.711+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f17d801c560 tx=0x7f17e003f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:43:06.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.711+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8009910 con 0x7f17d8047320 2024-09-10T03:43:06.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.711+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e800bd50 con 0x7f17d8047320 2024-09-10T03:43:06.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:06.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e800d3f0 con 0x7f17d8047320 2024-09-10T03:43:06.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:06.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:06.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:06.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:06.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:06.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:06.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:06.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:06.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:06.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:06.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:07.557 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:07.112+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:07.557 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:07.112+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:07.557 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:07.112+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:07.812 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:07.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:07.812 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:07.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:07.812 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:07.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:07.811+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:07.811+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:07.811+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:08.535 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:08.098+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:08.535 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:08.098+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:08.535 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:08.098+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:08.804 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:08.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:08.804 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:08.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:08.804 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:08.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:09.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:08.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:09.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:08.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:09.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:08.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:09.532 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:09.118+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:09.532 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:09.118+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:09.532 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:09.118+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:09.818 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:09.531+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:09.818 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:09.531+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:09.818 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:09.531+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:09.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:09.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:09.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:10.490 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:10.109+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:10.490 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:10.109+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:10.490 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:10.109+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:10.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:10.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:10.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:10.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:10.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:10.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:11.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:10.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:11.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:10.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:11.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:10.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:11.517 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:11.085+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:11.517 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:11.085+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:11.517 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:11.085+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:11.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.709+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:43:11.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.709+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:43:11.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.709+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f17d801c560 tx=0x7f17e003f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:43:11.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.709+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f17d801c560 tx=0x7f17e003f040 comp rx=0 tx=0).stop 2024-09-10T03:43:11.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.709+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:43:11.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.709+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:11.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.710+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:11.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.710+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d804c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:11.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.710+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8055110 con 0x7f17f0100840 2024-09-10T03:43:11.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.710+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8055290 con 0x7f17f0101a40 2024-09-10T03:43:11.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.710+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d8050580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:43:11.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.710+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d804c030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:43:11.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.710+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d804c030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:11.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.710+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17e0010a80 con 0x7f17f0100840 2024-09-10T03:43:11.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.711+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d8050580 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f17d801c390 tx=0x7f17e8009420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:43:11.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.711+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0050040 con 0x7f17f0100840 2024-09-10T03:43:11.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.711+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e001a400 con 0x7f17f0100840 2024-09-10T03:43:11.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:11.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0045270 con 0x7f17f0100840 2024-09-10T03:43:11.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:11.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:11.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:11.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:11.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:11.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:11.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:11.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:11.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:11.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:12.503 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:12.118+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:12.503 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:12.118+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:12.503 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:12.118+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:12.763 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:12.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:12.763 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:12.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:12.763 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:12.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:13.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:12.760+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:13.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:12.760+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:13.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:12.760+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:13.513 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:13.155+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:13.513 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:13.155+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:13.513 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:13.155+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:13.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:13.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:13.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:13.524+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:13.524+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:13.524+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:14.563 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:14.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:14.563 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:14.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:14.563 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:14.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:14.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:14.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:14.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:14.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:14.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:14.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:14.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:14.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:14.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:15.557 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:15.208+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:15.557 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:15.208+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:15.557 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:15.208+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:15.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:15.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:15.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:15.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:15.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:15.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:15.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:16.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.273+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.169:6828/1204620466, fault. 2024-09-10T03:43:16.275 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.273+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T03:43:16.537 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:16.189+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:16.538 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:16.189+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:16.538 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:16.189+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:16.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.709+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d8050580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:43:16.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.709+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d8050580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:43:16.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.709+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d8050580 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f17d801c390 tx=0x7f17e8009420 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:43:16.711 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.709+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d8050580 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f17d801c390 tx=0x7f17e8009420 comp rx=0 tx=0).stop 2024-09-10T03:43:16.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.709+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:43:16.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.709+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d8050580 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:16.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.709+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:16.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.709+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d804c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:16.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.709+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8055640 con 0x7f17d8047320 2024-09-10T03:43:16.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.709+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80557c0 con 0x7f17f0101a40 2024-09-10T03:43:16.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.709+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:43:16.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.710+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d804c030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:43:16.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.710+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d804c030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:16.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.710+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17e0010a80 con 0x7f17d8047320 2024-09-10T03:43:16.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.711+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f17e004f260 tx=0x7f17e0017fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:43:16.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.711+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0015390 con 0x7f17d8047320 2024-09-10T03:43:16.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.711+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0015560 con 0x7f17d8047320 2024-09-10T03:43:16.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:16.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0041c50 con 0x7f17d8047320 2024-09-10T03:43:16.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:16.739+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:16.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:16.739+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:16.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:16.739+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:16.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:16.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:16.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:17.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:17.192+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:17.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:17.192+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:17.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:17.192+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:18.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:17.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:18.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:17.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:18.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:17.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:18.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:17.583+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:17.583+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:17.583+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:18.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:18.142+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:18.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:18.142+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:18.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:18.142+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:18.673+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:18.673+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:18.673+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:19.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:18.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:18.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:18.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:19.147+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:19.147+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:19.147+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:19.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:19.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:19.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:20.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:19.604+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:19.604+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:19.604+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:20.572 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:20.194+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:20.572 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:20.194+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:20.572 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:20.194+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:20.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:20.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:20.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:20.719+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:20.719+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:20.719+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:21.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:21.229+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:21.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:21.229+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:21.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:21.229+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:21.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.710+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:43:21.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.710+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:43:21.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.710+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f17e004f260 tx=0x7f17e0017fb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:43:21.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.710+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f17e004f260 tx=0x7f17e0017fb0 comp rx=0 tx=0).stop 2024-09-10T03:43:21.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.710+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:43:21.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.710+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:21.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.710+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:21.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.710+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d804c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:21.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.710+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8055930 con 0x7f17f0100840 2024-09-10T03:43:21.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.710+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8055ab0 con 0x7f17f0101a40 2024-09-10T03:43:21.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.711+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d8050580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:43:21.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.711+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d804c030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:43:21.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.711+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d804c030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:21.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.711+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17e0010a80 con 0x7f17f0100840 2024-09-10T03:43:21.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.712+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d8050580 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f17e0001150 tx=0x7f17e001a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:43:21.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0061070 con 0x7f17f0100840 2024-09-10T03:43:21.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.713+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0047540 con 0x7f17f0100840 2024-09-10T03:43:21.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:21.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0050cd0 con 0x7f17f0100840 2024-09-10T03:43:21.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:21.673+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:21.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:21.673+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:21.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:21.673+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:21.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:21.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:21.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:22.506 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:22.252+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:22.506 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:22.252+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:22.506 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:22.252+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:22.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:22.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:22.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:22.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:22.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:22.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:22.505+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:22.505+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:22.505+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:23.484 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:23.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:23.484 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:23.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:23.484 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:23.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:23.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:23.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:23.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:23.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:23.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:23.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:24.521 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:24.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:24.521 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:24.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:24.521 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:24.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:24.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:24.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:24.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:24.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:24.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:24.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:24.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:24.520+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:24.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:24.520+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:24.520+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:25.500 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:25.155+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:25.500 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:25.155+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:25.500 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:25.155+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:25.692+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:25.692+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:25.692+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:25.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:25.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:25.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:26.497 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:26.179+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:26.497 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:26.179+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:26.497 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:26.179+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:26.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.710+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d8050580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T03:43:26.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.710+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d8050580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:43:26.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.710+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d8050580 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f17e0001150 tx=0x7f17e001a8c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:43:26.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.710+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d8050580 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f17e0001150 tx=0x7f17e001a8c0 comp rx=0 tx=0).stop 2024-09-10T03:43:26.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.711+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:43:26.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.711+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d8050580 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:26.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.711+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:26.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.711+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d804c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:26.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.711+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8055ac0 con 0x7f17d8047320 2024-09-10T03:43:26.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.711+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8055c40 con 0x7f17f0101a40 2024-09-10T03:43:26.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.711+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:43:26.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.712+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d804c030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:43:26.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.712+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d804c030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:26.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.712+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17e0010a80 con 0x7f17d8047320 2024-09-10T03:43:26.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.712+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f17e003f250 tx=0x7f17e000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:43:26.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028040 con 0x7f17d8047320 2024-09-10T03:43:26.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0015820 con 0x7f17d8047320 2024-09-10T03:43:26.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:26.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063080 con 0x7f17d8047320 2024-09-10T03:43:26.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:26.723+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:26.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:26.723+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:26.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:26.723+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:26.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:26.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:26.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:26.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:26.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:27.535 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:27.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:27.535 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:27.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:27.535 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:27.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:27.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:27.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:27.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:27.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:27.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:27.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:27.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:27.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:27.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:27.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:27.881+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.169:6828/1204620466, fault. 2024-09-10T03:43:27.882 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:27.881+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-10T03:43:28.522 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:28.165+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:28.522 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:28.165+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:28.522 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:28.165+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:28.520+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:28.520+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:28.520+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:28.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:28.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:28.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:29.479 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:29.181+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:29.479 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:29.181+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:29.479 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:29.181+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:29.743 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:29.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:29.743 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:29.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:29.743 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:29.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:29.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:29.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:29.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:30.498 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:30.201+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:30.499 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:30.201+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:30.499 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:30.201+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:30.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:30.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:30.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:30.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:30.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:30.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:30.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:30.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:31.207+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:31.207+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:31.207+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:31.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.711+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T03:43:31.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.711+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:43:31.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.711+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f17e003f250 tx=0x7f17e000b8b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:43:31.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.711+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f17e003f250 tx=0x7f17e000b8b0 comp rx=0 tx=0).stop 2024-09-10T03:43:31.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.711+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:43:31.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.711+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:31.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.711+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:31.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.711+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:31.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.711+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8055a10 con 0x7f17f0100840 2024-09-10T03:43:31.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.711+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8055b90 con 0x7f17f0101a40 2024-09-10T03:43:31.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.712+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:43:31.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.712+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:43:31.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.712+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:31.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.712+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17e0010a80 con 0x7f17f0100840 2024-09-10T03:43:31.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.713+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009a80 tx=0x7f17e0009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:43:31.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.713+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e800ce80 con 0x7f17f0100840 2024-09-10T03:43:31.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.713+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8049070 con 0x7f17f0100840 2024-09-10T03:43:31.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:31.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8015420 con 0x7f17f0100840 2024-09-10T03:43:31.791 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:31.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:31.791 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:31.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:31.791 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:31.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:32.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:31.790+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:32.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:31.790+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:32.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:31.790+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:32.481 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:32.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:32.481 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:32.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:32.481 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:32.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:32.760 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:32.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:32.760 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:32.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:32.761 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:32.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:32.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:32.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:32.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:33.513 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:33.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:33.513 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:33.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:33.513 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:33.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:33.514 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:33.471+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:33.514 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:33.471+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:33.514 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:33.471+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:34.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:33.797+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:34.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:33.797+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:34.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:33.797+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:34.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:34.458+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:34.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:34.458+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:34.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:34.458+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:34.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:34.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:34.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:35.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:34.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:35.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:34.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:35.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:34.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:35.465 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:35.207+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:35.466 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:35.207+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:35.466 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:35.207+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:35.739 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:35.464+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:35.740 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:35.464+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:35.740 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:35.464+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:36.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:35.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:36.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:35.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:36.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:35.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:36.439 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:36.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:36.439 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:36.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:36.439 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:36.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:36.709 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:36.438+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:36.709 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:36.438+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:36.709 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:36.438+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:36.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.711+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:43:36.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.711+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:43:36.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.711+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009a80 tx=0x7f17e0009190 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:43:36.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.711+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009a80 tx=0x7f17e0009190 comp rx=0 tx=0).stop 2024-09-10T03:43:36.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.711+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:43:36.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.711+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:36.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.711+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:36.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.711+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:36.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8055fd0 con 0x7f17d8047320 2024-09-10T03:43:36.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8056150 con 0x7f17f0101a40 2024-09-10T03:43:36.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.712+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:43:36.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.713+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:43:36.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.713+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:36.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.713+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17e0010a80 con 0x7f17d8047320 2024-09-10T03:43:36.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.713+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f17d800d710 tx=0x7f17e8004000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:43:36.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028040 con 0x7f17d8047320 2024-09-10T03:43:36.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0011630 con 0x7f17d8047320 2024-09-10T03:43:36.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:36.715+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e00159e0 con 0x7f17d8047320 2024-09-10T03:43:37.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:36.707+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:37.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:36.707+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:36.707+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:37.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:37.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:37.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:37.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:37.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:37.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:37.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:37.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:37.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:37.681+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:37.681+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:37.681+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:38.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:38.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:38.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:38.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:38.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:38.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:38.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:38.201+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:38.201+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:38.201+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:39.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:38.633+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:39.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:38.633+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:39.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:38.633+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:39.479 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:39.189+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:39.479 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:39.189+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:39.479 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:39.189+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:39.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:39.679+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:43:39.680 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:39.679+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-09-10T03:43:39.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:39.633+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:39.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:39.633+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:39.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:39.633+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:39.477+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:39.477+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:39.477+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:40.504 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:40.200+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:40.504 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:40.200+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:40.504 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:40.200+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:40.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:40.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:40.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:40.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:40.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:40.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:41.552 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:41.198+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:41.552 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:41.198+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:41.552 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:41.198+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:41.553 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:41.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:41.553 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:41.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:41.553 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:41.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:41.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.712+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T03:43:41.713 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.712+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:43:41.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.712+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f17d800d710 tx=0x7f17e8004000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:43:41.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.712+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f17d800d710 tx=0x7f17e8004000 comp rx=0 tx=0).stop 2024-09-10T03:43:41.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:43:41.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.712+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:41.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.712+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:41.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.712+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:41.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8055fd0 con 0x7f17f0100840 2024-09-10T03:43:41.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8056150 con 0x7f17f0101a40 2024-09-10T03:43:41.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.712+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:43:41.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.713+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:43:41.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.713+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:41.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.713+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17e0010a80 con 0x7f17f0100840 2024-09-10T03:43:41.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.714+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f17e80022f0 tx=0x7f17e001a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:43:41.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8014e00 con 0x7f17f0100840 2024-09-10T03:43:41.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.716+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b030 con 0x7f17f0100840 2024-09-10T03:43:41.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:41.716+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8015420 con 0x7f17f0100840 2024-09-10T03:43:41.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:41.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:41.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:41.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:41.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:41.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:42.513 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:42.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:42.514 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:42.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:42.514 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:42.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:42.620+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:42.620+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:42.620+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:42.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:42.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:42.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:42.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:43.514 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:43.147+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:43.514 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:43.147+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:43.514 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:43.147+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:43.528+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:43.528+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:43.528+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:43.571+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:43.571+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:43.571+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:44.503 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:44.135+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:44.504 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:44.135+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:44.504 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:44.135+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:44.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:44.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:44.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:44.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:44.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:44.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:44.524+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:44.524+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:44.524+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:45.537 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:45.173+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:45.538 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:45.173+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:45.538 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:45.173+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:45.824 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:45.536+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:45.824 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:45.536+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:45.824 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:45.536+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:45.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:45.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:45.825 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:45.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:45.825 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:45.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:46.541 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:46.145+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:46.541 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:46.145+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:46.541 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:46.145+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:46.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.712+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:43:46.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.712+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:43:46.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.712+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f17e80022f0 tx=0x7f17e001a8c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:43:46.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.712+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f17e80022f0 tx=0x7f17e001a8c0 comp rx=0 tx=0).stop 2024-09-10T03:43:46.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:43:46.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.712+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:46.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.712+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:46.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.712+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:46.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8056510 con 0x7f17d8047320 2024-09-10T03:43:46.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8056690 con 0x7f17f0101a40 2024-09-10T03:43:46.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.713+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:43:46.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.714+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:43:46.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.714+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:46.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.714+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17e0010a80 con 0x7f17d8047320 2024-09-10T03:43:46.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.714+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f17d802f220 tx=0x7f17e8007210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:43:46.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.715+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028030 con 0x7f17d8047320 2024-09-10T03:43:46.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0047370 con 0x7f17d8047320 2024-09-10T03:43:46.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:46.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063080 con 0x7f17d8047320 2024-09-10T03:43:46.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:46.539+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:46.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:46.539+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:46.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:46.539+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:46.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:46.558+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:46.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:46.558+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:46.558+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:47.533 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:47.142+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:47.533 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:47.142+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:47.533 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:47.142+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:47.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:47.532+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:47.532+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:47.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:47.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:47.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:48.488 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:48.156+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:48.489 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:48.156+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:48.489 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:48.156+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:48.483+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:48.483+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:48.483+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:48.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:48.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:48.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:48.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:48.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:48.896 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:48.894+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:43:48.897 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:48.894+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-09-10T03:43:49.532 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:49.184+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:49.532 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:49.184+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:49.533 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:49.184+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:49.530+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:49.530+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:49.530+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:49.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:49.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:49.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:50.492 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:50.225+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:50.492 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:50.225+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:50.492 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:50.225+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:50.490+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:50.490+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:50.490+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:50.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:50.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:50.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:50.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:51.538 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:51.271+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:51.538 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:51.271+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:51.538 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:51.271+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:51.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.712+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:43:51.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.712+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:43:51.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.712+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f17d802f220 tx=0x7f17e8007210 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:43:51.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.712+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f17d802f220 tx=0x7f17e8007210 comp rx=0 tx=0).stop 2024-09-10T03:43:51.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 msgr2=0x7f17d8049710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:43:51.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.712+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047320 0x7f17d8049710 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:51.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.712+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:51.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.712+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:51.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8056510 con 0x7f17f0100840 2024-09-10T03:43:51.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.712+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8056690 con 0x7f17f0101a40 2024-09-10T03:43:51.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.712+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:43:51.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.713+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:43:51.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.713+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:51.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.713+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17e0010a80 con 0x7f17f0100840 2024-09-10T03:43:51.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.713+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f17e8055da0 tx=0x7f17e000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:43:51.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.713+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8045420 con 0x7f17f0100840 2024-09-10T03:43:51.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.716+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17f0100840 2024-09-10T03:43:51.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:51.716+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8015420 con 0x7f17f0100840 2024-09-10T03:43:51.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:51.538+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:51.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:51.538+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:51.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:51.538+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:51.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:51.535+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:51.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:51.535+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:51.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:51.535+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:52.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:52.317+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:52.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:52.317+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:52.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:52.317+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:52.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:52.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:52.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:53.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:52.584+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:53.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:52.584+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:53.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:52.584+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:53.623 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:53.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:53.623 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:53.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:53.623 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:53.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:53.624 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:53.347+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:53.624 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:53.347+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:53.624 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:53.347+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:54.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:53.622+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:54.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:53.622+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:54.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:53.622+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:54.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:54.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:54.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:54.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:54.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:54.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:55.040 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:54.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:55.040 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:54.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:55.040 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:54.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:55.041 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:54.597+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:55.041 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:54.597+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:55.041 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:54.597+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:55.255+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:55.255+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:55.255+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:55.623+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:55.623+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:55.623+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:55.546+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:55.546+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:55.546+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:56.550 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:56.248+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:56.550 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:56.248+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:56.550 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:56.248+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:56.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.713+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:43:56.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.713+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:43:56.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.713+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f17e8055da0 tx=0x7f17e000b8b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:43:56.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.713+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f17e8055da0 tx=0x7f17e000b8b0 comp rx=0 tx=0).stop 2024-09-10T03:43:56.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.713+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:43:56.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.713+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:56.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.713+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047700 0x7f17d8056810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:56.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.713+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:43:56.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.713+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8056d50 con 0x7f17d8047700 2024-09-10T03:43:56.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.713+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8056ed0 con 0x7f17f0101a40 2024-09-10T03:43:56.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.713+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047700 0x7f17d8056810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:43:56.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.714+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:43:56.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.714+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:43:56.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.714+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17e0010a80 con 0x7f17d8047700 2024-09-10T03:43:56.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.714+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047700 0x7f17d8056810 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f17d8019d60 tx=0x7f17e8050fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:43:56.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c040 con 0x7f17d8047700 2024-09-10T03:43:56.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0015bb0 con 0x7f17d8047700 2024-09-10T03:43:56.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:43:56.716+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063020 con 0x7f17d8047700 2024-09-10T03:43:56.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:56.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:56.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:56.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:56.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:56.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:56.614+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:56.614+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:56.614+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:57.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:57.262+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:57.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:57.262+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:57.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:57.262+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:58.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:57.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:57.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:57.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:57.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:57.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:57.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:58.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:58.251+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:58.251+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:58.251+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:59.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:58.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:59.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:58.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:59.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:58.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:59.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:58.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:58.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:58.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:59.574 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:59.240+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:59.574 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:59.240+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:59.575 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:43:59.240+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:43:59.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:59.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:59.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:59.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:59.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:43:59.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:43:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:59.573+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:59.573+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:43:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:43:59.573+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:00.570 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:00.237+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:00.570 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:00.237+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:00.570 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:00.237+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:00.622+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:00.622+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:00.622+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:00.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:00.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:00.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:01.564 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:01.256+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:01.564 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:01.256+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:01.564 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:01.256+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:01.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.713+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047700 msgr2=0x7f17d8056810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:44:01.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.713+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047700 msgr2=0x7f17d8056810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:44:01.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.713+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047700 0x7f17d8056810 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f17d8019d60 tx=0x7f17e8050fa0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:44:01.714 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.713+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047700 0x7f17d8056810 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f17d8019d60 tx=0x7f17e8050fa0 comp rx=0 tx=0).stop 2024-09-10T03:44:01.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.713+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047700 msgr2=0x7f17d8056810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:44:01.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.713+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047700 0x7f17d8056810 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:01.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.713+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:01.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.713+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:01.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.713+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8056e00 con 0x7f17f0100840 2024-09-10T03:44:01.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.713+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8056f80 con 0x7f17f0101a40 2024-09-10T03:44:01.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.713+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:44:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.714+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:44:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.714+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.714+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17f0195cd0 con 0x7f17f0100840 2024-09-10T03:44:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.714+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f17d8009f70 tx=0x7f17e0009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:44:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.715+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e80048f0 con 0x7f17f0100840 2024-09-10T03:44:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.715+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17f0100840 2024-09-10T03:44:01.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:01.715+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8015b60 con 0x7f17f0100840 2024-09-10T03:44:01.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:01.563+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:01.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:01.563+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:01.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:01.563+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:01.647+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:01.647+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:01.647+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:02.579 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:02.244+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:02.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:02.244+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:02.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:02.244+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:02.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:02.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:02.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:02.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:02.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:02.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:02.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:02.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:03.516 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:03.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:03.516 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:03.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:03.516 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:03.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:03.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:03.637+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:03.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:03.637+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:03.637+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:03.594+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:03.594+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:03.594+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:04.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:04.255+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:44:04.256 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:04.255+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:44:04.579 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:04.226+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:04.579 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:04.226+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:04.579 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:04.226+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:04.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:04.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:04.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:04.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:04.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:04.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:05.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:05.243+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:05.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:05.243+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:05.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:05.243+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:06.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:05.597+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:06.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:05.597+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:06.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:05.597+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:05.708+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:05.708+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:05.708+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:06.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:06.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:06.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:06.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:06.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:06.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:06.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.713+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T03:44:06.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.713+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:44:06.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.713+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f17d8009f70 tx=0x7f17e0009190 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:44:06.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.713+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f17d8009f70 tx=0x7f17e0009190 comp rx=0 tx=0).stop 2024-09-10T03:44:06.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:44:06.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.714+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:06.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.714+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:06.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.714+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 0x7f17d8056e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:06.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80573f0 con 0x7f17d8056810 2024-09-10T03:44:06.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8057570 con 0x7f17f0101a40 2024-09-10T03:44:06.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.714+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 0x7f17d8056e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:44:06.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.714+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:44:06.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.714+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:06.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.714+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17f0195cd0 con 0x7f17d8056810 2024-09-10T03:44:06.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.715+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 0x7f17d8056e00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f17d80103c0 tx=0x7f17e8004800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:44:06.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.715+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055040 con 0x7f17d8056810 2024-09-10T03:44:06.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.716+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d8056810 2024-09-10T03:44:06.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:06.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c080 con 0x7f17d8056810 2024-09-10T03:44:07.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:06.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:07.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:06.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:07.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:06.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:06.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:06.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:06.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:07.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:07.234+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:07.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:07.234+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:07.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:07.234+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:07.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:07.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:07.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:07.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:07.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:07.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:08.214+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:08.214+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:08.214+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:09.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:08.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:09.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:08.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:09.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:08.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:09.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:08.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:08.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:08.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:09.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:09.217+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:09.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:09.217+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:09.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:09.217+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:09.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:09.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:09.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:09.689+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:09.689+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:09.689+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:10.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:10.173+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:10.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:10.173+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:10.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:10.173+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:11.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:10.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:11.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:10.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:11.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:10.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:10.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:10.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:10.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:11.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:11.220+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:11.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:11.220+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:11.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:11.220+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:11.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.714+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 msgr2=0x7f17d8056e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:44:11.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.714+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 msgr2=0x7f17d8056e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:44:11.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.714+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 0x7f17d8056e00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f17d80103c0 tx=0x7f17e8004800 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:44:11.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.714+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 0x7f17d8056e00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f17d80103c0 tx=0x7f17e8004800 comp rx=0 tx=0).stop 2024-09-10T03:44:11.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 msgr2=0x7f17d8056e00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:44:11.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.714+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 0x7f17d8056e00 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:11.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.714+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:11.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.714+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:11.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80573f0 con 0x7f17f0100840 2024-09-10T03:44:11.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8057570 con 0x7f17f0101a40 2024-09-10T03:44:11.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.714+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:44:11.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.715+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:44:11.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.715+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:11.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.715+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17f0195cd0 con 0x7f17f0100840 2024-09-10T03:44:11.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.716+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f17d8016490 tx=0x7f17e8000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:44:11.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.716+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e00172c0 con 0x7f17f0100840 2024-09-10T03:44:11.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.716+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e00117f0 con 0x7f17f0100840 2024-09-10T03:44:11.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:11.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17f0100840 2024-09-10T03:44:12.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:11.680+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:12.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:11.680+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:12.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:11.680+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:11.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:11.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:11.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:12.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:12.221+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:12.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:12.221+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:12.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:12.221+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:13.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:12.668+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:13.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:12.668+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:13.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:12.668+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:13.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:12.715+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:13.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:12.715+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:13.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:12.715+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:13.474 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:13.179+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:13.474 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:13.179+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:13.474 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:13.179+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:13.715+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:13.715+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:13.715+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:13.692+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:13.692+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:13.692+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:14.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:14.224+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:14.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:14.224+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:14.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:14.224+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:15.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:14.706+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:15.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:14.706+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:15.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:14.706+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:15.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:14.726+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:15.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:14.726+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:15.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:14.726+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:15.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:15.263+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:15.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:15.263+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:15.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:15.263+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:16.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:15.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:16.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:15.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:16.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:15.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:16.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:15.707+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:16.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:15.707+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:16.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:15.707+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:16.544 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:16.217+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:16.544 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:16.217+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:16.544 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:16.217+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:16.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.714+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:44:16.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.714+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:44:16.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.714+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f17d8016490 tx=0x7f17e8000f40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:44:16.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.714+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f17d8016490 tx=0x7f17e8000f40 comp rx=0 tx=0).stop 2024-09-10T03:44:16.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.715+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:44:16.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.715+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:16.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.715+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 0x7f17d8056dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:16.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.715+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:16.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.715+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8057300 con 0x7f17d8056810 2024-09-10T03:44:16.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.715+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8057480 con 0x7f17f0101a40 2024-09-10T03:44:16.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.715+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 0x7f17d8056dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:44:16.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.716+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:44:16.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.716+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:16.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.716+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17f0195cd0 con 0x7f17d8056810 2024-09-10T03:44:16.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.716+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 0x7f17d8056dc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f17e005a1f0 tx=0x7f17e0011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:44:16.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0061040 con 0x7f17d8056810 2024-09-10T03:44:16.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e004cd30 con 0x7f17d8056810 2024-09-10T03:44:16.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:16.718+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8056810 2024-09-10T03:44:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:16.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:16.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:16.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:16.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:16.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:16.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:16.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:16.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:16.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:17.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:17.198+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:17.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:17.198+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:17.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:17.198+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:18.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:17.687+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:18.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:17.687+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:18.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:17.687+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:18.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:17.704+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:18.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:17.704+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:18.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:17.704+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:18.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:18.188+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:18.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:18.188+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:18.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:18.188+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:18.737+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:18.737+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:18.737+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:19.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:18.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:18.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:18.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:19.180+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:19.180+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:19.180+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:19.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:19.615+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:44:19.616 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:19.615+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:44:20.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:19.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:19.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:19.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:19.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:19.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:19.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:20.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:20.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:20.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:20.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:20.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:20.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:21.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:20.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:21.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:20.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:21.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:20.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:21.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:20.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:20.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:20.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:21.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:21.125+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:21.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:21.125+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:21.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:21.125+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:21.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.714+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 msgr2=0x7f17d8056dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:44:21.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.715+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 msgr2=0x7f17d8056dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:44:21.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.715+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 0x7f17d8056dc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f17e005a1f0 tx=0x7f17e0011e80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:44:21.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.715+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 0x7f17d8056dc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f17e005a1f0 tx=0x7f17e0011e80 comp rx=0 tx=0).stop 2024-09-10T03:44:21.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.715+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 msgr2=0x7f17d8056dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:44:21.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.715+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056810 0x7f17d8056dc0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:21.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.715+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:21.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.715+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:21.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.715+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8057470 con 0x7f17f0100840 2024-09-10T03:44:21.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.715+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80575f0 con 0x7f17f0101a40 2024-09-10T03:44:21.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.715+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:44:21.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.716+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:44:21.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.716+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:21.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.716+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17f0195cd0 con 0x7f17f0100840 2024-09-10T03:44:21.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.716+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f17e001a810 tx=0x7f17e002ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:44:21.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028040 con 0x7f17f0100840 2024-09-10T03:44:21.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e004cd30 con 0x7f17f0100840 2024-09-10T03:44:21.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:21.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17f0100840 2024-09-10T03:44:22.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:21.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:22.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:21.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:22.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:21.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:21.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:21.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:21.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:22.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:22.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:22.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:22.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:22.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:22.725+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:22.725+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:22.725+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:22.701+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:22.701+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:22.701+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:23.517 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:23.111+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:23.518 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:23.111+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:23.518 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:23.111+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:23.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:23.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:23.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:23.737+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:23.737+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:23.737+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:24.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:24.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:24.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:25.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:24.693+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:25.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:24.693+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:25.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:24.694+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:25.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:24.732+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:25.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:24.732+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:25.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:24.732+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:25.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:25.093+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:25.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:25.093+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:25.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:25.093+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:26.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:25.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:26.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:25.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:26.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:25.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:25.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:25.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:25.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:26.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:26.093+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:26.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:26.093+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:26.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:26.093+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:26.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.714+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:44:26.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.714+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:44:26.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.714+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f17e001a810 tx=0x7f17e002ef90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:44:26.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.714+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f17e001a810 tx=0x7f17e002ef90 comp rx=0 tx=0).stop 2024-09-10T03:44:26.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 msgr2=0x7f17d804c030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:44:26.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.714+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17f0100840 0x7f17d804c030 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:26.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.714+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:26.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.714+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:26.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80570c0 con 0x7f17d8056ce0 2024-09-10T03:44:26.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.714+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80579b0 con 0x7f17f0101a40 2024-09-10T03:44:26.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.715+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:44:26.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.715+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:44:26.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.715+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:26.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.715+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8057b30 con 0x7f17d8056ce0 2024-09-10T03:44:26.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.716+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f17e001a810 tx=0x7f17e0047370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:44:26.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e00151c0 con 0x7f17d8056ce0 2024-09-10T03:44:26.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.719+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0045e80 con 0x7f17d8056ce0 2024-09-10T03:44:26.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:26.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063050 con 0x7f17d8056ce0 2024-09-10T03:44:27.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:26.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:27.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:26.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:27.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:26.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:27.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:26.676+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:26.676+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:26.676+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:27.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:27.100+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:27.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:27.100+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:27.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:27.100+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:28.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:27.659+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:28.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:27.659+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:28.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:27.659+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:28.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:27.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:28.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:27.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:28.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:27.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:28.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:28.111+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:28.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:28.111+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:28.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:28.111+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:29.066 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:28.633+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:29.066 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:28.633+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:29.066 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:28.633+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:29.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:28.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:29.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:28.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:29.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:28.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:29.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:29.063+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:29.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:29.063+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:29.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:29.063+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:30.073 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:29.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:30.073 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:29.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:30.073 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:29.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:30.073 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:29.649+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:30.073 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:29.649+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:30.073 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:29.649+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:30.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:30.072+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:30.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:30.072+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:30.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:30.072+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:31.028 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:30.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:31.029 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:30.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:31.029 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:30.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:31.029 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:30.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:31.029 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:30.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:31.029 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:30.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:31.027+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:31.027+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:31.027+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:31.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.715+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:44:31.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.715+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:44:31.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.715+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f17e001a810 tx=0x7f17e0047370 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:44:31.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.715+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f17e001a810 tx=0x7f17e0047370 comp rx=0 tx=0).stop 2024-09-10T03:44:31.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.715+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:44:31.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.715+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:31.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.715+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 0x7f17d8057d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:31.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.715+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:31.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.715+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80579b0 con 0x7f17d804a020 2024-09-10T03:44:31.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.715+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80570c0 con 0x7f17f0101a40 2024-09-10T03:44:31.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.716+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 0x7f17d8057d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:44:31.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.716+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:44:31.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.716+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:31.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.716+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8057b30 con 0x7f17d804a020 2024-09-10T03:44:31.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.717+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 0x7f17d8057d00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f17e0039000 tx=0x7f17e000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:44:31.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e004a420 con 0x7f17d804a020 2024-09-10T03:44:31.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.718+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0015dc0 con 0x7f17d804a020 2024-09-10T03:44:31.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:31.719+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d804a020 2024-09-10T03:44:32.055 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:31.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:32.056 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:31.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:32.056 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:31.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:32.056 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:31.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:32.056 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:31.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:32.056 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:31.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:32.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:32.054+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:32.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:32.054+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:32.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:32.054+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:33.021 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:32.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:33.021 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:32.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:33.021 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:32.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:33.021 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:32.637+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:33.021 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:32.637+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:33.022 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:32.637+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:33.020+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:33.020+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:33.020+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:33.619+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:33.619+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:33.619+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:33.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:33.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:33.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:33.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:33.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:33.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:34.964 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:34.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:34.964 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:34.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:34.964 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:34.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:34.965 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:34.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:34.965 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:34.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:34.965 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:34.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:34.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:34.975+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:44:34.976 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:34.975+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:44:35.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:34.963+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:35.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:34.963+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:35.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:34.963+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:35.986 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:35.641+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:35.986 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:35.641+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:35.986 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:35.641+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:35.987 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:35.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:35.987 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:35.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:35.987 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:35.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:35.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:35.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:35.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:36.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.716+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 msgr2=0x7f17d8057d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:44:36.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.716+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 msgr2=0x7f17d8057d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:44:36.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.716+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 0x7f17d8057d00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f17e0039000 tx=0x7f17e000b040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:44:36.717 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.716+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 0x7f17d8057d00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f17e0039000 tx=0x7f17e000b040 comp rx=0 tx=0).stop 2024-09-10T03:44:36.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.716+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 msgr2=0x7f17d8057d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:44:36.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.716+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 0x7f17d8057d00 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:36.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.716+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:36.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.716+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:36.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.716+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80570c0 con 0x7f17d8056ce0 2024-09-10T03:44:36.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.716+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8057240 con 0x7f17f0101a40 2024-09-10T03:44:36.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.716+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:44:36.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.717+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:44:36.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.717+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:36.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.717+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8057b30 con 0x7f17d8056ce0 2024-09-10T03:44:36.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.717+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f17e8008fa0 tx=0x7f17e001bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:44:36.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.718+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8032020 con 0x7f17d8056ce0 2024-09-10T03:44:36.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.718+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d8056ce0 2024-09-10T03:44:36.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:36.718+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c080 con 0x7f17d8056ce0 2024-09-10T03:44:37.000 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:36.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:37.000 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:36.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:37.000 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:36.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:37.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:36.702+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:37.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:36.702+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:37.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:36.702+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:37.001 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:44:36.715+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:44:37.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:36.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:37.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:36.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:37.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:36.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:37.659+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:37.659+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:37.659+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:37.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:37.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:37.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:37.951+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:37.951+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:37.951+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:38.928 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:38.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:38.929 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:38.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:38.929 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:38.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:38.929 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:38.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:38.929 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:38.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:38.929 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:38.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:38.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:38.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:38.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:39.961 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:39.612+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:39.962 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:39.612+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:39.962 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:39.612+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:39.962 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:39.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:39.962 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:39.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:39.962 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:39.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:39.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:39.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:39.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:40.998 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:40.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:40.999 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:40.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:40.999 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:40.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:40.999 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:40.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:40.999 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:40.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:40.999 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:40.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:40.997+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:40.997+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:40.997+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:41.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.716+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:44:41.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.716+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:44:41.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.716+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f17e8008fa0 tx=0x7f17e001bde0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:44:41.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.716+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f17e8008fa0 tx=0x7f17e001bde0 comp rx=0 tx=0).stop 2024-09-10T03:44:41.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.716+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:44:41.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.716+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:41.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.716+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 0x7f17d8057d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:41.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.716+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:41.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.716+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80570c0 con 0x7f17d804a020 2024-09-10T03:44:41.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.716+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8057240 con 0x7f17f0101a40 2024-09-10T03:44:41.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.717+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 0x7f17d8057d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:44:41.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.717+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:44:41.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.717+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:41.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.717+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8057b30 con 0x7f17d804a020 2024-09-10T03:44:41.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.718+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 0x7f17d8057d00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f17e000b3f0 tx=0x7f17e8001f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:44:41.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.719+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0061070 con 0x7f17d804a020 2024-09-10T03:44:41.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.719+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e00117f0 con 0x7f17d804a020 2024-09-10T03:44:41.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:41.721+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d804a020 2024-09-10T03:44:42.012 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:44:41.716+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:44:42.013 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:41.652+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:42.013 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:41.652+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:42.013 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:41.652+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:42.013 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:41.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:42.013 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:41.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:42.013 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:41.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:42.011+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:42.011+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:42.011+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:42.685+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:42.685+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:42.685+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:42.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:42.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:42.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:42.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:43.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:43.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:43.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:43.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:43.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:43.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:43.662+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:43.662+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:43.662+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:43.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:43.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:43.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:44.005+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:44.005+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:44.005+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:44.705+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:44.705+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:44.705+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:44.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:44.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:44.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:44.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:44.981+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:44.981+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:44.981+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:46.017 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:45.661+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:46.017 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:45.661+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:46.017 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:45.661+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:46.018 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:45.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:46.018 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:45.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:46.018 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:45.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:46.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:46.016+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:46.016+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:46.016+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:46.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.717+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 msgr2=0x7f17d8057d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:44:46.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.717+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 msgr2=0x7f17d8057d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:44:46.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.717+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 0x7f17d8057d00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f17e000b3f0 tx=0x7f17e8001f80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:44:46.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.717+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 0x7f17d8057d00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f17e000b3f0 tx=0x7f17e8001f80 comp rx=0 tx=0).stop 2024-09-10T03:44:46.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 msgr2=0x7f17d8057d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:44:46.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.717+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a020 0x7f17d8057d00 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:46.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.717+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:46.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.717+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:46.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80570c0 con 0x7f17d8056ce0 2024-09-10T03:44:46.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8057240 con 0x7f17f0101a40 2024-09-10T03:44:46.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.717+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:44:46.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.718+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:44:46.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.718+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:46.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.718+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8057b30 con 0x7f17d8056ce0 2024-09-10T03:44:46.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.718+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f17e80022f0 tx=0x7f17e00159e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:44:46.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.719+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8032030 con 0x7f17d8056ce0 2024-09-10T03:44:46.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.719+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d8056ce0 2024-09-10T03:44:46.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:46.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17d8056ce0 2024-09-10T03:44:46.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:44:46.716+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:44:46.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:46.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:46.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:46.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:46.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:46.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:46.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:47.329 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:47.012+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:47.329 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:47.012+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:47.329 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:47.012+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:47.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:47.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:47.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:47.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:47.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:47.994 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:47.649+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:47.994 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:47.649+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:47.994 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:47.649+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:48.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:47.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:48.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:47.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:48.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:47.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:48.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:48.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:48.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:48.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:48.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:48.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:49.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:48.949+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:49.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:48.949+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:49.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:48.949+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:49.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:49.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:49.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:49.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:49.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:49.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:49.987+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:49.987+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:49.987+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:50.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:50.399+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:44:50.400 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:50.399+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:44:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:50.587+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:50.587+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:50.587+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:50.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:50.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:50.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:50.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:51.035+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:51.035+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:51.035+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:51.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.717+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:44:51.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.717+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:44:51.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.717+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f17e80022f0 tx=0x7f17e00159e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:44:51.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.717+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f17e80022f0 tx=0x7f17e00159e0 comp rx=0 tx=0).stop 2024-09-10T03:44:51.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:44:51.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.717+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057470 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:51.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.717+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a400 0x7f17d804a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:51.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.717+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:51.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8057170 con 0x7f17d804a400 2024-09-10T03:44:51.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804ad20 con 0x7f17f0101a40 2024-09-10T03:44:51.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.718+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a400 0x7f17d804a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:44:51.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.718+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:44:51.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.718+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:51.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.718+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8057b30 con 0x7f17d804a400 2024-09-10T03:44:51.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.719+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a400 0x7f17d804a7e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f17e00227b0 tx=0x7f17e800cfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:44:51.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0061040 con 0x7f17d804a400 2024-09-10T03:44:51.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e003bec0 con 0x7f17d804a400 2024-09-10T03:44:51.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:51.721+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d804a400 2024-09-10T03:44:51.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:51.590+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:51.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:51.590+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:51.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:51.590+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:51.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:51.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:51.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:51.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:44:51.717+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:44:52.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:51.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:52.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:51.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:52.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:51.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:52.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:52.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:52.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:52.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:52.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:52.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:52.619+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:52.619+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:52.619+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:53.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:53.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:53.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:53.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:53.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:53.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:53.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:53.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:53.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:54.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:54.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:54.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:54.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:54.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:54.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:54.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:54.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:54.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:55.063+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:55.063+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:55.063+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:55.629+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:55.629+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:55.629+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:55.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:55.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:55.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:56.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:56.032+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:56.032+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:56.718 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.717+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a400 msgr2=0x7f17d804a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:44:56.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.717+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a400 msgr2=0x7f17d804a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:44:56.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.717+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a400 0x7f17d804a7e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f17e00227b0 tx=0x7f17e800cfb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:44:56.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.717+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a400 0x7f17d804a7e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f17e00227b0 tx=0x7f17e800cfb0 comp rx=0 tx=0).stop 2024-09-10T03:44:56.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a400 msgr2=0x7f17d804a7e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:44:56.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.717+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a400 0x7f17d804a7e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:56.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.717+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:56.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.717+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d80570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:44:56.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80576b0 con 0x7f17d8056ce0 2024-09-10T03:44:56.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8057830 con 0x7f17f0101a40 2024-09-10T03:44:56.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.718+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d80570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:44:56.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.718+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:44:56.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.718+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:44:56.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.718+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8057b30 con 0x7f17d8056ce0 2024-09-10T03:44:56.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.719+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d80570c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f17e800c940 tx=0x7f17e005d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:44:56.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8032040 con 0x7f17d8056ce0 2024-09-10T03:44:56.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e80159b0 con 0x7f17d8056ce0 2024-09-10T03:44:56.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:44:56.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17d8056ce0 2024-09-10T03:44:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:56.601+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:56.601+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:56.601+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:56.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:56.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:56.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:56.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:56.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:44:56.717+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 45 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:44:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:57.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:57.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:57.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:57.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:57.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:57.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:57.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:57.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:57.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:57.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:57.540+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:57.540+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:57.540+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:58.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:58.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:58.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:58.580+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:58.580+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:58.580+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:58.537+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:58.537+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:58.537+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:59.046+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:59.046+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:44:59.046+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:44:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:59.574+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:59.574+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:44:59.574+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:44:59.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:59.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:59.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:44:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:44:59.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:00.016+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:00.016+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:00.016+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:00.514+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:00.514+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:00.514+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:00.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:00.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:00.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:00.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:01.001+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:01.001+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:01.001+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:01.719 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.717+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d80570c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:45:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.717+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d80570c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:45:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.717+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d80570c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f17e800c940 tx=0x7f17e005d040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:45:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.717+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d80570c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f17e800c940 tx=0x7f17e005d040 comp rx=0 tx=0).stop 2024-09-10T03:45:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d80570c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:45:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.717+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d80570c0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.717+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.717+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804adf0 con 0x7f17d80576b0 2024-09-10T03:45:01.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804af70 con 0x7f17f0101a40 2024-09-10T03:45:01.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.718+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:45:01.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.718+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:45:01.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.718+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:01.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.718+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8057b30 con 0x7f17d80576b0 2024-09-10T03:45:01.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.719+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f17e804e7e0 tx=0x7f17e802c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:45:01.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028070 con 0x7f17d80576b0 2024-09-10T03:45:01.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0043e90 con 0x7f17d80576b0 2024-09-10T03:45:01.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:01.722+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d80576b0 2024-09-10T03:45:01.832 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:45:01.717+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 62 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:45:01.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:01.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:01.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:01.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:01.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:01.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:01.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:01.477+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:01.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:01.477+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:01.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:01.477+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:02.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:01.977+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:02.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:01.977+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:02.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:01.977+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:02.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:02.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:02.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:02.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:02.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:02.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:02.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:02.466+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:02.466+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:02.466+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:02.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:02.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:02.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:03.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:03.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:03.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:03.481+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:03.481+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:03.481+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:03.923+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:03.923+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:03.923+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:04.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:04.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:04.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:04.499+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:04.499+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:04.499+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:04.970+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:04.970+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:04.970+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:05.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:05.758+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:45:05.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:05.758+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:45:05.760 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:05.716+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:05.760 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:05.716+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:05.760 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:05.716+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:05.761 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:05.495+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:05.761 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:05.495+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:05.761 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:05.495+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:06.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:06.004+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:06.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:06.004+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:06.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:06.004+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:06.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.717+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 msgr2=0x7f17d804a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:45:06.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.717+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 msgr2=0x7f17d804a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:45:06.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.717+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f17e804e7e0 tx=0x7f17e802c000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:45:06.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.717+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f17e804e7e0 tx=0x7f17e802c000 comp rx=0 tx=0).stop 2024-09-10T03:45:06.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.717+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 msgr2=0x7f17d804a7e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:45:06.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.717+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:06.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.718+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d80570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:06.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.718+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:06.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.718+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804af80 con 0x7f17d8056ce0 2024-09-10T03:45:06.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.718+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804b100 con 0x7f17f0101a40 2024-09-10T03:45:06.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.718+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d80570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:45:06.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.719+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:45:06.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.719+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:06.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.719+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d804b280 con 0x7f17d8056ce0 2024-09-10T03:45:06.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.719+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d80570c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f17e000b3f0 tx=0x7f17e000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:45:06.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028070 con 0x7f17d8056ce0 2024-09-10T03:45:06.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.721+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e001fe80 con 0x7f17d8056ce0 2024-09-10T03:45:06.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:06.722+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063050 con 0x7f17d8056ce0 2024-09-10T03:45:06.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:45:06.717+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 79 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:45:06.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:06.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:06.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:06.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:06.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:06.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:06.672+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:06.672+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:06.672+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:07.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:07.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:07.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:07.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:07.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:07.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:07.488+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:07.488+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:07.488+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:07.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:07.694+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:07.694+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:07.694+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:08.052+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:08.052+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:08.052+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:08.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:08.522+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:08.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:08.522+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:08.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:08.522+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:08.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:08.721+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:08.721+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:08.721+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:09.063+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:09.063+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:09.063+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:09.745 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:09.477+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:09.745 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:09.477+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:09.745 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:09.477+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:10.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:09.743+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:10.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:09.743+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:10.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:09.743+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:10.478 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:10.101+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:10.478 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:10.101+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:10.478 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:10.101+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:10.788 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:10.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:10.788 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:10.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:10.788 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:10.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:11.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:10.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:11.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:10.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:11.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:10.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:11.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:11.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:11.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:11.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.719+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d80570c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:45:11.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.719+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d80570c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:45:11.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.719+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d80570c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f17e000b3f0 tx=0x7f17e000b040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:45:11.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.719+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d80570c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f17e000b3f0 tx=0x7f17e000b040 comp rx=0 tx=0).stop 2024-09-10T03:45:11.720 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.719+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d80570c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:45:11.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.719+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d80570c0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:11.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.719+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:11.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.719+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:11.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.719+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804b0c0 con 0x7f17d80576b0 2024-09-10T03:45:11.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.719+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804b450 con 0x7f17f0101a40 2024-09-10T03:45:11.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.719+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:45:11.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.719+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:45:11.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.720+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:11.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.720+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d804b5d0 con 0x7f17d80576b0 2024-09-10T03:45:11.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.721+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f17e0041010 tx=0x7f17e0009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:45:11.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.721+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028070 con 0x7f17d80576b0 2024-09-10T03:45:11.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.723+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e00159e0 con 0x7f17d80576b0 2024-09-10T03:45:11.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:11.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063050 con 0x7f17d80576b0 2024-09-10T03:45:11.774 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:45:11.718+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 99 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:45:11.774 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:11.519+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:11.775 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:11.519+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:11.775 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:11.519+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:12.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:11.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:12.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:11.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:12.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:11.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:12.500 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:12.111+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:12.500 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:12.111+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:12.500 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:12.111+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:12.809 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:12.499+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:12.809 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:12.499+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:12.809 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:12.499+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:13.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:12.808+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:13.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:12.808+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:13.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:12.808+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:13.466 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:13.160+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:13.466 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:13.160+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:13.466 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:13.160+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:13.464+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:13.464+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:13.464+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:14.197 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:13.835+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:14.197 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:13.835+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:14.197 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:13.835+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:14.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:14.441+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:14.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:14.441+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:14.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:14.441+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:14.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:14.196+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:14.196+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:14.196+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:15.245 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:14.848+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:15.245 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:14.848+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:15.245 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:14.848+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:15.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:15.456+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:15.456+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:15.456+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:15.244+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:15.244+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:15.244+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:16.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:15.826+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:16.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:15.826+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:16.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:15.826+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:16.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:16.436+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:16.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:16.436+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:16.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:16.436+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:16.290+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:16.290+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:16.290+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:16.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.719+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 msgr2=0x7f17d804a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:45:16.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.719+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 msgr2=0x7f17d804a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:45:16.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.719+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f17e0041010 tx=0x7f17e0009860 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:45:16.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.719+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f17e0041010 tx=0x7f17e0009860 comp rx=0 tx=0).stop 2024-09-10T03:45:16.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.719+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 msgr2=0x7f17d804a7e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:45:16.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.719+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:16.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.719+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:16.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.719+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:16.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.719+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804b520 con 0x7f17d8056ce0 2024-09-10T03:45:16.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.719+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804b6a0 con 0x7f17f0101a40 2024-09-10T03:45:16.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.720+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:45:16.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.720+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:45:16.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.720+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:16.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.720+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d804b820 con 0x7f17d8056ce0 2024-09-10T03:45:16.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.721+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057160 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f17d80108e0 tx=0x7f17e000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:45:16.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.721+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055050 con 0x7f17d8056ce0 2024-09-10T03:45:16.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.723+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d8056ce0 2024-09-10T03:45:16.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:16.723+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c040 con 0x7f17d8056ce0 2024-09-10T03:45:17.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:45:16.719+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 116 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:45:17.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:16.855+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:17.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:16.855+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:17.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:16.855+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:17.483 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:17.245+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:17.483 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:17.245+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:17.483 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:17.245+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:17.481+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:17.481+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:17.481+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:18.286 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:17.860+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:18.286 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:17.860+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:18.286 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:17.860+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:18.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:18.491+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:18.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:18.491+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:18.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:18.491+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:18.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:18.285+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:18.285+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:18.285+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:19.240 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:18.868+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:19.240 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:18.868+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:19.240 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:18.868+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:19.530 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:19.239+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:19.530 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:19.239+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:19.530 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:19.239+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:19.827 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:19.529+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:19.827 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:19.529+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:19.827 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:19.529+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:20.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:19.825+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:20.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:19.825+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:20.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:19.825+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:20.549 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:20.199+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:20.549 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:20.199+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:20.549 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:20.199+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:20.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:20.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:20.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:21.120 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.119+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:45:21.120 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.119+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:45:21.120 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:20.847+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:21.121 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:20.847+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:21.121 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:20.847+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:21.559 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:21.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:21.559 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:21.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:21.559 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:21.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:21.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.719+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:45:21.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.719+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:45:21.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.719+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057160 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f17d80108e0 tx=0x7f17e000c9c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:45:21.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.720+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057160 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f17d80108e0 tx=0x7f17e000c9c0 comp rx=0 tx=0).stop 2024-09-10T03:45:21.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:45:21.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.720+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057160 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:21.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.720+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:21.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.720+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:21.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804b5f0 con 0x7f17d80576b0 2024-09-10T03:45:21.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804b9f0 con 0x7f17f0101a40 2024-09-10T03:45:21.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.720+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:45:21.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.721+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:45:21.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.721+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:21.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.721+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d804b820 con 0x7f17d80576b0 2024-09-10T03:45:21.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.721+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f17d8010850 tx=0x7f17e8000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:45:21.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055040 con 0x7f17d80576b0 2024-09-10T03:45:21.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d80576b0 2024-09-10T03:45:21.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:21.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d80576b0 2024-09-10T03:45:21.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:21.558+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:21.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:21.558+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:21.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:21.558+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:21.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:45:21.719+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 133 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:45:22.122 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:21.857+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:22.122 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:21.857+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:22.122 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:21.857+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:22.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:22.121+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:22.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:22.121+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:22.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:22.121+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:22.818 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:22.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:22.818 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:22.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:22.818 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:22.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:23.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:22.816+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:23.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:22.816+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:23.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:22.816+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:23.440 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:23.128+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:23.440 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:23.128+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:23.440 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:23.128+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:23.809 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:23.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:23.809 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:23.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:23.809 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:23.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:24.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:23.808+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:24.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:23.808+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:24.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:23.808+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:24.550 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:24.134+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:24.550 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:24.134+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:24.550 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:24.134+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:24.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:24.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:24.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:24.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:24.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:24.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:24.761+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:24.761+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:24.761+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:25.524 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:25.128+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:25.524 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:25.128+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:25.524 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:25.128+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:25.783 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:25.522+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:25.783 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:25.522+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:25.783 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:25.522+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:26.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:25.782+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:25.782+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:25.782+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:26.495 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:26.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:26.495 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:26.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:26.495 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:26.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:26.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.720+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 msgr2=0x7f17d804a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:45:26.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.720+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 msgr2=0x7f17d804a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:45:26.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.720+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f17d8010850 tx=0x7f17e8000f40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:45:26.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.720+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f17d8010850 tx=0x7f17e8000f40 comp rx=0 tx=0).stop 2024-09-10T03:45:26.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 msgr2=0x7f17d804a7e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:45:26.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.720+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:26.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.720+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:26.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.720+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:26.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804b5f0 con 0x7f17d8056ce0 2024-09-10T03:45:26.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804b9f0 con 0x7f17f0101a40 2024-09-10T03:45:26.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.721+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:45:26.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.721+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:45:26.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.721+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:26.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.721+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d804b820 con 0x7f17d8056ce0 2024-09-10T03:45:26.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.722+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f17d8019d60 tx=0x7f17e804ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:45:26.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.723+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8032030 con 0x7f17d8056ce0 2024-09-10T03:45:26.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e804c370 con 0x7f17d8056ce0 2024-09-10T03:45:26.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:26.725+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17d8056ce0 2024-09-10T03:45:26.766 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:45:26.720+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 150 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:45:26.766 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:26.494+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:26.766 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:26.494+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:26.766 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:26.494+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:27.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:26.765+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:27.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:26.765+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:27.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:26.765+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:27.494 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:27.100+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:27.494 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:27.100+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:27.494 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:27.100+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:27.810 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:27.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:27.810 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:27.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:27.810 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:27.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:28.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:27.809+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:28.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:27.809+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:28.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:27.809+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:28.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:28.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:28.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:28.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:28.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:28.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:28.812 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:28.516+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:28.813 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:28.516+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:28.813 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:28.516+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:29.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:28.811+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:29.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:28.811+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:28.811+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:29.518 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:29.082+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:29.518 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:29.082+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:29.519 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:29.082+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:29.803 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:29.517+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:29.803 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:29.517+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:29.804 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:29.517+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:30.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:30.049+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:30.049+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:30.049+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:29.802+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:29.802+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:30.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:29.802+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:30.799 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:30.494+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:30.800 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:30.494+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:30.800 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:30.494+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:31.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:31.017+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:31.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:31.017+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:31.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:31.017+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:31.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:30.798+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:30.798+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:30.798+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:31.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.720+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:45:31.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.720+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:45:31.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.720+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f17d8019d60 tx=0x7f17e804ee80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:45:31.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.720+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f17d8019d60 tx=0x7f17e804ee80 comp rx=0 tx=0).stop 2024-09-10T03:45:31.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.721+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:45:31.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.721+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:31.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.721+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:31.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.721+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:31.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.721+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804b5d0 con 0x7f17d80576b0 2024-09-10T03:45:31.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.721+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804b9f0 con 0x7f17f0101a40 2024-09-10T03:45:31.723 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:31.453+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:31.723 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:31.453+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:31.723 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:31.453+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:31.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.721+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:45:31.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.721+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:45:31.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.721+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:31.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.721+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d804b820 con 0x7f17d80576b0 2024-09-10T03:45:31.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.722+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f17d802f250 tx=0x7f17e80094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:45:31.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.723+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055040 con 0x7f17d80576b0 2024-09-10T03:45:31.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.723+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e804c370 con 0x7f17d80576b0 2024-09-10T03:45:31.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:31.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c050 con 0x7f17d80576b0 2024-09-10T03:45:32.011 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:31.801+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:32.011 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:31.801+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:32.011 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:31.801+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:32.011 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:45:31.720+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 170 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:45:32.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:32.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:32.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:32.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:32.757 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:32.490+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:32.758 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:32.490+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:32.758 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:32.490+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:33.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:32.968+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:33.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:32.968+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:33.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:32.968+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:33.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:32.969+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:32.756+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:32.756+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:32.756+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:32.757+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.24321.0:5177 3.8 3:125d909c:::1000000471e.00000000:head [write 0~4329 in=4329b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:33.491+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:33.491+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:33.491+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:33.493+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:33.720+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:33.720+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:33.720+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:33.721+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:33.993+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:33.993+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:33.993+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:33.993+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:34.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:34.519+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:34.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:34.519+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:34.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:34.519+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:34.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:34.519+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:34.726+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:34.726+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:34.726+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:34.727+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:34.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:34.854+0000 7f17de7fc640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f17d40034c0 con 0x7f17d80576b0 2024-09-10T03:45:35.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:34.946+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:35.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:34.946+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:35.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:34.946+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:35.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:34.946+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:35.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:35.707+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:35.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:35.707+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:35.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:35.707+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:35.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:35.708+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:35.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:35.514+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:35.834 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:35.514+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:35.834 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:35.514+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:35.834 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:35.514+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:35.855 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:35.855+0000 7f17de7fc640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f17d4005820 con 0x7f17d80576b0 2024-09-10T03:45:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:35.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:35.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:35.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:35.992+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:36.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.479+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:45:36.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.479+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:45:36.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.721+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 msgr2=0x7f17d804a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:45:36.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.721+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 msgr2=0x7f17d804a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:45:36.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.721+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f17d802f250 tx=0x7f17e80094c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:45:36.722 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.721+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f17d802f250 tx=0x7f17e80094c0 comp rx=0 tx=0).stop 2024-09-10T03:45:36.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.721+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 msgr2=0x7f17d804a7e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:45:36.723 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:36.470+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:36.723 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:36.470+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:36.723 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:36.470+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:36.723 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:36.470+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:36.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.721+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d80576b0 0x7f17d804a7e0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:36.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.721+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:36.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.721+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:36.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.721+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804b5d0 con 0x7f17d8056ce0 2024-09-10T03:45:36.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.721+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804b9f0 con 0x7f17f0101a40 2024-09-10T03:45:36.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.721+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:45:36.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.722+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:45:36.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.722+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:36.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.722+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d804b820 con 0x7f17d8056ce0 2024-09-10T03:45:36.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.722+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f17e800bf80 tx=0x7f17e0009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:45:36.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.723+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8032100 con 0x7f17d8056ce0 2024-09-10T03:45:36.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.723+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b020 con 0x7f17d8056ce0 2024-09-10T03:45:36.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:36.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17d8056ce0 2024-09-10T03:45:37.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:45:36.721+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 190 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:45:37.039 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:36.757+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:37.039 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:36.757+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:37.039 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:36.757+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:37.039 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:36.758+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:37.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:37.037+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:37.037+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:37.037+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:37.038+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:37.480+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:37.480+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:37.480+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:37.481+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:37.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:37.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:37.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:37.712+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:37.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:37.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:37.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:37.993+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:38.745 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:38.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:38.746 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:38.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:38.746 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:38.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:38.746 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:38.697+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:38.746 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:38.462+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:38.746 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:38.462+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:38.746 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:38.462+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:38.747 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:38.463+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:38.998+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:38.998+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:38.998+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:38.999+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:39.707 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:39.429+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:39.708 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:39.429+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:39.708 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:39.429+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:39.708 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:39.430+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:39.990 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:39.705+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:39.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:39.705+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:39.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:39.705+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:39.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:39.705+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:39.988+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:39.988+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:39.988+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:39.989+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:40.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:40.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:40.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:40.667+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:40.425+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:40.425+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:40.425+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:40.426+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:41.013+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:41.013+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:41.013+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:41.013+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:41.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.720+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:45:41.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.720+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:45:41.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.720+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f17e800bf80 tx=0x7f17e0009930 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:45:41.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.720+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f17e800bf80 tx=0x7f17e0009930 comp rx=0 tx=0).stop 2024-09-10T03:45:41.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:45:41.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.720+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:41.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.720+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:41.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.720+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 0x7f17d804b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:41.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804b5d0 con 0x7f17d804a7e0 2024-09-10T03:45:41.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.720+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804bf30 con 0x7f17f0101a40 2024-09-10T03:45:41.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.721+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 0x7f17d804b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:45:41.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.721+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:45:41.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.721+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:41.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.721+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d804c0b0 con 0x7f17d804a7e0 2024-09-10T03:45:41.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.722+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 0x7f17d804b9f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f17e803e4b0 tx=0x7f17e8037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:45:41.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.722+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8009de0 con 0x7f17d804a7e0 2024-09-10T03:45:41.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.722+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d804a7e0 2024-09-10T03:45:41.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:41.723+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17d804a7e0 2024-09-10T03:45:41.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:45:41.720+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 207 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:45:41.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:41.665+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:41.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:41.665+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:41.665+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:41.665+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:41.473+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:41.473+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:41.473+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:41.474+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:42.014+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:42.014+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:42.014+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:42.015+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:42.453+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:42.453+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:42.453+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:42.453+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:42.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:42.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:42.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:42.623+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:43.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:43.041+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:43.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:43.041+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:43.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:43.041+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:43.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:43.042+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:43.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:43.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:43.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:43.585+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:43.437+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:43.437+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:43.437+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:43.437+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:44.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:44.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:44.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:44.076+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:44.410+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:44.410+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:44.410+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:44.411+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:44.570+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:44.570+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:44.570+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:44.570+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:45.048+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:45.048+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:45.048+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:45.049+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:45.696 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:45.531+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:45.696 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:45.531+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:45.697 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:45.531+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:45.697 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:45.532+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:45.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:45.392+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:45.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:45.392+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:45.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:45.392+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:45.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:45.393+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:46.369 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:46.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:46.369 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:46.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:46.369 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:46.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:46.369 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:46.097+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:46.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.722+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 msgr2=0x7f17d804b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:45:46.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.722+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 msgr2=0x7f17d804b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:45:46.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.722+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 0x7f17d804b9f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f17e803e4b0 tx=0x7f17e8037000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:45:46.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.722+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 0x7f17d804b9f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f17e803e4b0 tx=0x7f17e8037000 comp rx=0 tx=0).stop 2024-09-10T03:45:46.723 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.722+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 msgr2=0x7f17d804b9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:45:46.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.722+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 0x7f17d804b9f0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:46.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.722+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:46.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.722+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:46.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.722+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804b5d0 con 0x7f17d8056ce0 2024-09-10T03:45:46.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.722+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804c280 con 0x7f17f0101a40 2024-09-10T03:45:46.724 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:46.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:46.724 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:46.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:46.724 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:46.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:46.724 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:46.562+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:46.725 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:46.367+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:46.725 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:46.367+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:46.725 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:46.367+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:46.725 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:46.368+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:46.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.723+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:45:46.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.724+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:45:46.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.724+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:46.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.724+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d804c0b0 con 0x7f17d8056ce0 2024-09-10T03:45:46.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.724+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f17d800ffa0 tx=0x7f17e803e690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:45:46.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.725+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e00476f0 con 0x7f17d8056ce0 2024-09-10T03:45:46.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.726+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e001ce70 con 0x7f17d8056ce0 2024-09-10T03:45:46.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:46.726+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0024d10 con 0x7f17d8056ce0 2024-09-10T03:45:47.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:45:46.722+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 229 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:45:47.394 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:47.131+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:47.394 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:47.131+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:47.394 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:47.131+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:47.395 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:47.132+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:47.393+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:47.393+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:47.393+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:47.394+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:47.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:47.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:47.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:47.586+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:48.426 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:48.140+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:48.426 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:48.140+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:48.426 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:48.140+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:48.427 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:48.141+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:48.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:48.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:48.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:48.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:48.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:48.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:48.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:48.590+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:48.769 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:48.425+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:48.769 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:48.425+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:48.770 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:48.425+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:48.770 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:48.426+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:49.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:49.377+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:49.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:49.377+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:49.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:49.377+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:49.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:49.377+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:49.137+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:49.137+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:49.137+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:49.138+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:50.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:49.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:50.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:49.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:50.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:49.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:50.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:49.582+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:50.391 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:50.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:50.392 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:50.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:50.392 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:50.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:50.392 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:50.117+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:50.390+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:50.390+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:50.390+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:50.391+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:50.565+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:50.565+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:50.565+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:50.566+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:51.381 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:51.110+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:51.382 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:51.110+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:51.382 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:51.110+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:51.382 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:51.111+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:51.723 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:51.380+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:51.724 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:51.380+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:51.724 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:51.380+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:51.724 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:51.381+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:51.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.722+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:45:51.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.722+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:45:51.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.722+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f17d800ffa0 tx=0x7f17e803e690 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:45:51.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.722+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f17d800ffa0 tx=0x7f17e803e690 comp rx=0 tx=0).stop 2024-09-10T03:45:51.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.722+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:45:51.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.722+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:51.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.722+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:51.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.722+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 0x7f17d804b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:51.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.722+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804b0c0 con 0x7f17d804a7e0 2024-09-10T03:45:51.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.722+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804b5d0 con 0x7f17f0101a40 2024-09-10T03:45:51.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.723+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 0x7f17d804b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:45:51.726 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:51.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:51.726 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:51.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:51.726 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:51.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:51.726 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:51.527+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:51.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.723+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:45:51.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.723+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:51.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.723+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d804b280 con 0x7f17d804a7e0 2024-09-10T03:45:51.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.724+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 0x7f17d804b9f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f17e801fff0 tx=0x7f17e00549f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:45:51.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.725+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055040 con 0x7f17d804a7e0 2024-09-10T03:45:51.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.725+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d804a7e0 2024-09-10T03:45:51.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.726+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d804a7e0 2024-09-10T03:45:51.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.839+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:45:51.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:51.839+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:45:52.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:45:51.722+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 246 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:45:52.407 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:52.138+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:52.407 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:52.138+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:52.407 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:52.138+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:52.407 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:52.138+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:52.405+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:52.405+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:52.405+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:52.406+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:52.564+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:52.564+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:52.564+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:52.565+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:53.523 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:53.390+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:53.523 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:53.390+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:53.523 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:53.390+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:53.523 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:53.391+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:53.524 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:53.154+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:53.524 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:53.154+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:53.524 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:53.154+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:53.524 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:53.154+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:53.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:53.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:53.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:53.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:53.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:53.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:53.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:53.570+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:54.105+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:54.105+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:54.105+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:54.106+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:54.431+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:54.431+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:54.431+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:54.431+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:54.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:54.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:54.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:54.585+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:55.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:55.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:55.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:55.081+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:55.446+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:55.446+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:55.446+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:55.446+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:55.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:55.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:55.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:55.629+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:56.048+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:56.048+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:56.048+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:56.049+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:56.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.722+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 msgr2=0x7f17d804b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T03:45:56.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.722+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 msgr2=0x7f17d804b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:45:56.724 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.722+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 0x7f17d804b9f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f17e801fff0 tx=0x7f17e00549f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:45:56.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.722+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 0x7f17d804b9f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f17e801fff0 tx=0x7f17e00549f0 comp rx=0 tx=0).stop 2024-09-10T03:45:56.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.722+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 msgr2=0x7f17d804b9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:45:56.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.722+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804a7e0 0x7f17d804b9f0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:56.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.722+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:56.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.723+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:45:56.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.723+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804bf30 con 0x7f17d8056ce0 2024-09-10T03:45:56.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.723+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804c0b0 con 0x7f17f0101a40 2024-09-10T03:45:56.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.723+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:45:56.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.723+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:45:56.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.723+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:45:56.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.723+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d804b280 con 0x7f17d8056ce0 2024-09-10T03:45:56.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.724+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f17e804df90 tx=0x7f17e8042330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:45:56.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0061040 con 0x7f17d8056ce0 2024-09-10T03:45:56.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e001ee70 con 0x7f17d8056ce0 2024-09-10T03:45:56.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:45:56.725+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063050 con 0x7f17d8056ce0 2024-09-10T03:45:56.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:45:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:45:56.723+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 263 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:45:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:56.489+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:56.489+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:56.489+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:56.490+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:56.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:56.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:56.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:56.634+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:57.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:57.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:57.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:57.065+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:57.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:57.468+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:57.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:57.468+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:57.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:57.468+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:57.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:57.469+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:57.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:57.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:57.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:57.683+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:58.024+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:58.024+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:58.024+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:58.025+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:58.456+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:58.456+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:58.456+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:58.457+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:58.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:58.691+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:58.691+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:58.691+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:58.692+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:45:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:59.030+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:59.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:59.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:45:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:45:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:45:59.031+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:59.712 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:59.461+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:59.712 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:59.461+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:59.712 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:59.461+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:45:59.713 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:45:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:45:59.461+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:45:59.713 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:59.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:59.713 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:59.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:59.713 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:59.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:45:59.713 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:45:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:45:59.711+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:00.035+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:00.035+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:00.035+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:00.036+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:00.435+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:00.435+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:00.435+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:00.436+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:00.672+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:00.672+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:00.672+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:00.673+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:01.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:01.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:01.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:01.075+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.723+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:46:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.723+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:46:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.723+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f17e804df90 tx=0x7f17e8042330 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:46:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.723+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f17e804df90 tx=0x7f17e8042330 comp rx=0 tx=0).stop 2024-09-10T03:46:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.723+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:46:01.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.723+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.723+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:01.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.723+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:01.726 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:01.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:01.726 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:01.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:01.726 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:01.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:01.726 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:01.408+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:01.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:01.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:01.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:01.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:01.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:01.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:01.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:01.634+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.723+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d804b9f0 2024-09-10T03:46:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.723+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:46:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.724+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:46:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.725+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:46:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.725+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.725+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d804b280 con 0x7f17d804b9f0 2024-09-10T03:46:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.725+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f17e0005f60 tx=0x7f17e0011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:46:01.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.726+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028020 con 0x7f17d804b9f0 2024-09-10T03:46:01.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.727+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e001fe80 con 0x7f17d804b9f0 2024-09-10T03:46:01.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:01.727+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d804b9f0 2024-09-10T03:46:02.056 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:46:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:46:01.723+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 280 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:46:02.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:02.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:02.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:02.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:02.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:02.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:02.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:02.056+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:02.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:02.619+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:02.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:02.619+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:02.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:02.619+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:02.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:02.620+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:02.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:02.401+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:02.401+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:02.401+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:02.401+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:03.021+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:03.021+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:03.021+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:03.022+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:03.650 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:03.363+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:03.651 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:03.363+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:03.651 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:03.363+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:03.651 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:03.364+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:04.032 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:03.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:04.032 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:03.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:04.032 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:03.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:04.032 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:03.650+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:04.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:04.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:04.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:04.032+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:04.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:04.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:04.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:04.634+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:04.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:04.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:04.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:04.390+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:05.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:05.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:05.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:05.035+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:05.425+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:05.425+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:05.425+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:05.426+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:05.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:05.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:05.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:05.637+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:06.077+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:06.077+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:06.077+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:06.078+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:06.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.723+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:46:06.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.723+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:46:06.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.723+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f17e0005f60 tx=0x7f17e0011e80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:46:06.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.724+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f17e0005f60 tx=0x7f17e0011e80 comp rx=0 tx=0).stop 2024-09-10T03:46:06.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:46:06.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.724+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:06.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.724+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:06.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.724+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:06.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d8056ce0 2024-09-10T03:46:06.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:46:06.726 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:06.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:06.726 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:06.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:06.726 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:06.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:06.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:06.672+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:06.727 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:06.461+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:06.727 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:06.461+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:06.727 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:06.461+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:06.727 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:06.462+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:06.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.725+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:46:06.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.726+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:46:06.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.726+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:06.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.726+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d805a010 con 0x7f17d8056ce0 2024-09-10T03:46:06.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.726+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f17e0047510 tx=0x7f17e001a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:46:06.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.727+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e00159e0 con 0x7f17d8056ce0 2024-09-10T03:46:06.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.728+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0037e90 con 0x7f17d8056ce0 2024-09-10T03:46:06.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:06.729+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8056ce0 2024-09-10T03:46:07.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:46:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:46:06.723+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 297 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:46:07.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:07.200+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:46:07.201 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:07.200+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:46:07.424 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:07.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:07.424 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:07.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:07.424 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:07.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:07.424 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:07.104+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:07.423+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:07.423+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:07.423+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:07.423+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:07.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:07.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:07.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:07.645+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:08.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:08.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:08.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:08.067+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:08.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:08.472+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:08.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:08.472+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:08.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:08.472+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:08.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:08.472+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:08.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:08.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:08.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:08.600+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:09.443 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:09.107+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:09.443 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:09.107+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:09.443 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:09.107+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:09.443 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:09.108+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:09.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:09.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:09.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:09.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:09.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:09.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:09.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:09.615+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:09.442+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:09.442+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:09.442+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:09.442+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:10.461 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:10.109+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:10.461 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:10.109+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:10.462 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:10.109+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:10.462 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:10.110+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:10.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:10.460+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:10.460+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:10.460+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:10.461+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:10.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:10.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:10.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:10.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:10.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:10.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:10.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:10.625+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:11.469 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:11.122+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:11.469 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:11.122+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:11.469 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:11.122+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:11.469 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:11.122+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:11.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.724+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:46:11.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.724+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:46:11.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.724+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f17e0047510 tx=0x7f17e001a8c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:46:11.725 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.724+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f17e0047510 tx=0x7f17e001a8c0 comp rx=0 tx=0).stop 2024-09-10T03:46:11.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:46:11.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.724+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:11.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.724+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:11.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.724+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:11.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d804b9f0 2024-09-10T03:46:11.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:46:11.726 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:11.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:11.726 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:11.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:11.726 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:11.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:11.726 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:11.635+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:11.727 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:11.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:11.727 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:11.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:11.727 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:11.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:11.727 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:11.468+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:11.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.724+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:46:11.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.725+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:46:11.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.725+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:11.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.725+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d805a010 con 0x7f17d804b9f0 2024-09-10T03:46:11.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.725+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f17e0008000 tx=0x7f17e0045930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:46:11.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.726+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0061040 con 0x7f17d804b9f0 2024-09-10T03:46:11.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.726+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0014e40 con 0x7f17d804b9f0 2024-09-10T03:46:11.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:11.727+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d804b9f0 2024-09-10T03:46:12.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:46:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:46:11.724+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 315 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:46:12.488 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:12.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:12.488 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:12.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:12.488 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:12.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:12.488 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:12.109+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:12.486+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:12.486+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:12.486+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:12.487+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:12.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:12.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:12.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:12.588+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:13.345 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:13.081+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:13.345 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:13.081+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:13.345 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:13.081+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:13.345 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:13.081+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:13.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:13.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:13.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:13.603+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:13.523+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:13.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:13.523+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:13.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:13.523+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:13.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:13.523+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:14.061+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:14.061+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:14.061+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:14.062+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:14.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:14.489+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:14.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:14.489+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:14.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:14.489+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:14.489+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:14.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:14.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:14.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:14.591+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:15.036+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:15.036+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:15.036+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:15.037+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:15.638 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:15.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:15.639 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:15.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:15.639 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:15.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:15.639 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:15.488+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:16.012 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:15.636+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:16.012 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:15.636+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:16.012 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:15.636+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:16.012 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:15.637+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:16.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:16.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:16.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:16.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:16.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:16.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:16.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:16.010+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:16.682 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:16.522+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:16.682 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:16.522+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:16.682 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:16.522+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:16.682 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:16.523+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:16.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.723+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:46:16.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.723+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:46:16.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.723+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f17e0008000 tx=0x7f17e0045930 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:46:16.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.723+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f17e0008000 tx=0x7f17e0045930 comp rx=0 tx=0).stop 2024-09-10T03:46:16.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.723+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:46:16.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.723+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:16.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.723+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:16.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.723+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:16.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.723+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d8056ce0 2024-09-10T03:46:16.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.723+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:46:16.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.724+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:46:16.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.724+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:46:16.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.724+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:16.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.724+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d805a010 con 0x7f17d8056ce0 2024-09-10T03:46:16.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.724+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f17e802bf80 tx=0x7f17e0011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:46:16.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055040 con 0x7f17d8056ce0 2024-09-10T03:46:16.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d8056ce0 2024-09-10T03:46:16.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:16.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d8056ce0 2024-09-10T03:46:16.993 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:46:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:46:16.723+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 332 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:46:16.993 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:16.680+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:16.993 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:16.680+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:16.993 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:16.680+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:16.993 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:16.680+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:17.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:16.990+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:17.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:16.990+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:17.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:16.990+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:17.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:16.991+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:17.533+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:17.841 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:17.533+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:17.841 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:17.533+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:17.842 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:17.534+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:17.842 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:17.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:17.842 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:17.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:17.842 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:17.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:17.842 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:17.648+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:17.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:17.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:17.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:18.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:17.992+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:18.511+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:18.511+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:18.511+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:18.511+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:18.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:18.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:18.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:18.644+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:19.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:19.032+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:19.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:19.032+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:19.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:19.032+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:19.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:19.032+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:19.503+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:19.503+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:19.503+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:19.504+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:19.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:19.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:19.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:19.606+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:20.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:20.070+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:20.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:20.070+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:20.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:20.070+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:20.070+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:20.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:20.538+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:20.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:20.538+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:20.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:20.538+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:20.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:20.539+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:20.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:20.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:20.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:20.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:20.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:20.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:20.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:20.578+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:21.502 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:21.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:21.502 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:21.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:21.502 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:21.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:21.503 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:21.109+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:21.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.724+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:46:21.726 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.724+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:46:21.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.724+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f17e802bf80 tx=0x7f17e0011e80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:46:21.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.724+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f17e802bf80 tx=0x7f17e0011e80 comp rx=0 tx=0).stop 2024-09-10T03:46:21.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.724+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 msgr2=0x7f17d8057100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:46:21.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.725+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8056ce0 0x7f17d8057100 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:21.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.725+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:21.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.725+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:21.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.725+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d804b9f0 2024-09-10T03:46:21.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.725+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:46:21.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.725+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:46:21.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.725+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:46:21.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.725+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:21.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.726+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d805a1e0 con 0x7f17d804b9f0 2024-09-10T03:46:21.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.726+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f17d801a700 tx=0x7f17e80468f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:46:21.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.726+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8003c00 con 0x7f17d804b9f0 2024-09-10T03:46:21.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.727+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d804b9f0 2024-09-10T03:46:21.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:21.727+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d804b9f0 2024-09-10T03:46:21.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:46:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:46:21.725+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 349 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:46:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:21.500+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:21.500+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:21.500+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:21.501+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:21.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:21.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:21.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:21.558+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:22.455 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:22.120+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:22.455 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:22.120+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:22.455 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:22.120+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:22.456 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:22.120+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:22.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:22.559+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:46:22.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:22.559+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:46:22.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:22.454+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:22.454+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:22.454+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:22.455+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:22.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:22.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:22.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:22.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:22.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:22.567+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:23.414 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:23.130+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:23.414 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:23.130+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:23.414 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:23.130+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:23.414 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:23.131+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:23.413+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:23.413+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:23.413+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:23.413+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:23.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:23.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:23.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:23.587+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:24.390 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:24.131+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:24.391 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:24.131+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:24.391 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:24.131+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:24.391 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:24.132+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:24.669 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:24.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:24.670 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:24.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:24.670 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:24.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:24.670 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:24.390+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:24.670 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:24.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:24.670 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:24.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:24.670 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:24.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:24.670 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:24.545+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:25.515 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:25.360+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:25.516 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:25.360+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:25.516 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:25.360+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:25.516 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:25.361+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:25.516 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:25.161+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:25.516 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:25.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:25.516 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:25.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:25.516 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:25.162+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:25.514+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:25.514+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:25.514+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:25.515+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:26.506 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:26.337+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:26.506 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:26.337+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:26.506 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:26.337+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:26.506 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:26.338+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:26.507 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:26.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:26.507 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:26.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:26.507 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:26.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:26.507 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:26.149+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.725+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:46:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.725+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:46:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.725+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f17d801a700 tx=0x7f17e80468f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:46:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.725+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f17d801a700 tx=0x7f17e80468f0 comp rx=0 tx=0).stop 2024-09-10T03:46:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.725+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:46:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.725+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804b9f0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.725+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.725+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8057100 0x7f17d805a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.725+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d8057100 2024-09-10T03:46:26.727 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.725+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:46:26.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.725+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8057100 0x7f17d805a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:46:26.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.726+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:46:26.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.726+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:26.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.726+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d805a1e0 con 0x7f17d8057100 2024-09-10T03:46:26.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.727+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8057100 0x7f17d805a3b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f17d800ae30 tx=0x7f17e8042330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:46:26.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.729+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8042e10 con 0x7f17d8057100 2024-09-10T03:46:26.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.729+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8009e10 con 0x7f17d8057100 2024-09-10T03:46:26.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:26.729+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d8057100 2024-09-10T03:46:26.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:46:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:46:26.725+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 366 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:46:26.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:26.505+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:26.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:26.505+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:26.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:26.505+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:26.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:26.506+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:27.536 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:27.384+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:27.536 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:27.384+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:27.536 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:27.384+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:27.536 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:27.385+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:27.536 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:27.195+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:27.537 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:27.195+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:27.537 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:27.195+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:27.537 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:27.196+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:27.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:27.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:27.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:27.535+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:28.550 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:28.364+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:28.550 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:28.364+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:28.550 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:28.364+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:28.550 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:28.364+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:28.551 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:28.176+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:28.551 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:28.176+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:28.551 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:28.176+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:28.551 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:28.177+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:28.549+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:28.549+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:28.549+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:28.550+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:29.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:29.377+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:29.377+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:29.377+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:29.378+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:29.166+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:29.166+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:29.166+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:29.166+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:30.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:29.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:30.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:29.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:30.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:29.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:30.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:29.595+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:30.339+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:30.339+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:30.339+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:30.339+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:30.130+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:30.130+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:30.130+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:30.131+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:31.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:30.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:31.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:30.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:31.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:30.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:30.592+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:31.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:31.351+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:31.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:31.351+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:31.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:31.351+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:31.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:31.352+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:31.107+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:31.107+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:31.107+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:31.107+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:31.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.725+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8057100 msgr2=0x7f17d805a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:46:31.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.725+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8057100 msgr2=0x7f17d805a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:46:31.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.726+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8057100 0x7f17d805a3b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f17d800ae30 tx=0x7f17e8042330 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:46:31.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.726+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8057100 0x7f17d805a3b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f17d800ae30 tx=0x7f17e8042330 comp rx=0 tx=0).stop 2024-09-10T03:46:31.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.726+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8057100 msgr2=0x7f17d805a3b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:46:31.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.726+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8057100 0x7f17d805a3b0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:31.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.726+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 0x7f17d805a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:31.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.726+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:31.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.726+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80563a0 con 0x7f17d804bdd0 2024-09-10T03:46:31.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.726+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:46:31.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.726+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 0x7f17d805a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:46:31.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.727+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:46:31.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.727+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:31.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.727+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d805a1e0 con 0x7f17d804bdd0 2024-09-10T03:46:31.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.727+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 0x7f17d805a8f0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f17d8007550 tx=0x7f17e804e850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:46:31.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.728+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c070 con 0x7f17d804bdd0 2024-09-10T03:46:31.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.728+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e002ad30 con 0x7f17d804bdd0 2024-09-10T03:46:31.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:31.729+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0018310 con 0x7f17d804bdd0 2024-09-10T03:46:32.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:46:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:46:31.726+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 383 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:46:32.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:31.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:32.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:31.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:32.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:31.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:32.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:31.586+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:32.376 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:32.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:32.377 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:32.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:32.377 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:32.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:32.377 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:32.081+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:32.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:32.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:32.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:32.589+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:32.375+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:32.375+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:32.375+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:32.376+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:33.526 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:33.332+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:33.526 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:33.332+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:33.526 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:33.332+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:33.526 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:33.333+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:33.526 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:33.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:33.526 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:33.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:33.527 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:33.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:33.527 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:33.118+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:33.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:33.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:33.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:33.623+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:34.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:34.289+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:34.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:34.289+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:34.289+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:34.289+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:34.113+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:34.113+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:34.113+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:34.113+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:34.873 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:34.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:34.873 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:34.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:34.873 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:34.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:34.873 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:34.671+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:35.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:35.240+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:35.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:35.240+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:35.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:35.240+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:35.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:35.241+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:35.141+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:35.141+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:35.141+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:35.142+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:36.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:35.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:36.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:35.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:36.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:35.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:36.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:35.685+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:36.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:36.242+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:36.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:36.242+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:36.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:36.242+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:36.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:36.243+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:36.113+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:36.113+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:36.113+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:36.114+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:36.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.726+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 msgr2=0x7f17d805a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:46:36.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.726+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 msgr2=0x7f17d805a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:46:36.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.726+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 0x7f17d805a8f0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f17d8007550 tx=0x7f17e804e850 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:46:36.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.726+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 0x7f17d805a8f0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f17d8007550 tx=0x7f17e804e850 comp rx=0 tx=0).stop 2024-09-10T03:46:36.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.726+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 msgr2=0x7f17d805a8f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:46:36.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.726+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 0x7f17d805a8f0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:36.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.726+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:36.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.726+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a1e0 0x7f17d805ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:36.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.726+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804c1b0 con 0x7f17d805a1e0 2024-09-10T03:46:36.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.726+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:46:36.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.727+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a1e0 0x7f17d805ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:46:36.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.727+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:46:36.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.727+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:36.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.727+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d804b280 con 0x7f17d805a1e0 2024-09-10T03:46:36.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.728+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a1e0 0x7f17d805ae30 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f17e8021020 tx=0x7f17e001a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:46:36.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.729+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8003c00 con 0x7f17d805a1e0 2024-09-10T03:46:36.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.729+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d805a1e0 2024-09-10T03:46:36.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:36.730+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d805a1e0 2024-09-10T03:46:37.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:46:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:46:36.726+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 400 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:46:37.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:36.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:37.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:36.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:37.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:36.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:37.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:36.667+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:37.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:37.289+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:37.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:37.289+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:37.289+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:37.290+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:37.131+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:37.131+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:37.131+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:37.132+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:37.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:37.920+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:46:37.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:37.920+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:46:38.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:37.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:38.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:37.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:38.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:37.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:38.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:37.679+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:38.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:38.098+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:38.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:38.098+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:38.098+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:38.099+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:38.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:38.267+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:38.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:38.267+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:38.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:38.267+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:38.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:38.268+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:39.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:38.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:39.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:38.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:39.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:38.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:39.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:38.635+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:39.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:39.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:39.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:39.056+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:39.262+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:39.262+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:39.262+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:39.263+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:39.982 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:39.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:39.982 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:39.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:39.982 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:39.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:39.982 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:39.602+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:40.252 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:40.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:40.252 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:40.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:40.252 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:40.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:40.252 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:40.075+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:40.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:40.250+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:40.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:40.250+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:40.250+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:40.251+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:41.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:40.642+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:41.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:40.642+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:41.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:40.642+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:41.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:40.643+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:41.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:41.248+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:41.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:41.248+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:41.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:41.248+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:41.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:41.249+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:41.113+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:41.113+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:41.113+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:41.114+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:41.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.726+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a1e0 msgr2=0x7f17d805ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:46:41.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.727+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a1e0 msgr2=0x7f17d805ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:46:41.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.727+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a1e0 0x7f17d805ae30 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f17e8021020 tx=0x7f17e001a8c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:46:41.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.727+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a1e0 0x7f17d805ae30 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f17e8021020 tx=0x7f17e001a8c0 comp rx=0 tx=0).stop 2024-09-10T03:46:41.728 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.727+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a1e0 msgr2=0x7f17d805ae30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:46:41.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.727+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a1e0 0x7f17d805ae30 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:41.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.727+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:41.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.727+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 0x7f17d805a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:41.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.727+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804c1b0 con 0x7f17d804bdd0 2024-09-10T03:46:41.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.727+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:46:41.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.727+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 0x7f17d805a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:46:41.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.728+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:46:41.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.728+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:41.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.728+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d804b280 con 0x7f17d804bdd0 2024-09-10T03:46:41.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.729+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 0x7f17d805a5e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f17e8020fc0 tx=0x7f17e8042fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:46:41.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.730+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055070 con 0x7f17d804bdd0 2024-09-10T03:46:41.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.731+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d804bdd0 2024-09-10T03:46:41.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:41.731+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d804bdd0 2024-09-10T03:46:42.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:46:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:46:41.726+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 417 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:46:42.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:41.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:42.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:41.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:41.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:41.669+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:42.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:42.234+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:42.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:42.234+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:42.234+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:42.235+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:42.129+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:42.129+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:42.129+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:42.129+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:43.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:42.621+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:43.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:42.621+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:42.621+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:42.621+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:43.527 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:43.094+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:43.527 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:43.094+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:43.527 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:43.094+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:43.527 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:43.095+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:43.527 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:43.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:43.527 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:43.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:43.528 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:43.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:43.528 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:43.230+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:43.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:43.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:43.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:43.625+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:44.127+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:44.127+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:44.127+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:44.127+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:44.208+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:44.208+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:44.208+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:44.209+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:45.076 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:44.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:45.076 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:44.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:45.076 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:44.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:45.076 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:44.655+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:45.236+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:45.236+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:45.236+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:45.237+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:45.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:45.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:45.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:45.079+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:46.034 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:45.655+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:46.034 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:45.655+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:46.034 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:45.655+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:46.035 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:45.656+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:46.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:46.241+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:46.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:46.241+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:46.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:46.241+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:46.242+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:46.033+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:46.033+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:46.033+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:46.034+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:46.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.727+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 msgr2=0x7f17d805a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:46:46.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.727+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 msgr2=0x7f17d805a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:46:46.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.727+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 0x7f17d805a5e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f17e8020fc0 tx=0x7f17e8042fb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:46:46.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.727+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 0x7f17d805a5e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f17e8020fc0 tx=0x7f17e8042fb0 comp rx=0 tx=0).stop 2024-09-10T03:46:46.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.727+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 msgr2=0x7f17d805a5e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:46:46.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.727+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d804bdd0 0x7f17d805a5e0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:46.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.727+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:46.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.727+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:46.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.727+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804c1b0 con 0x7f17d8047ae0 2024-09-10T03:46:46.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.727+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:46:46.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.728+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:46:46.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.728+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:46:46.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.728+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:46.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.728+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048400 con 0x7f17d8047ae0 2024-09-10T03:46:46.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.728+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f17e8020ff0 tx=0x7f17e802c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:46:46.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.730+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055070 con 0x7f17d8047ae0 2024-09-10T03:46:46.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.730+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d8047ae0 2024-09-10T03:46:46.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:46.732+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d8047ae0 2024-09-10T03:46:47.004 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:46.642+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:47.004 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:46.642+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:47.005 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:46.642+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:47.005 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:46.643+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:47.005 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:46:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:46:46.727+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 434 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:46:47.259 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:47.003+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:47.259 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:47.003+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:47.259 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:47.003+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:47.259 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:47.004+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:47.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:47.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:47.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:47.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:47.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:47.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:47.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:47.258+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:48.020 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:47.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:48.021 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:47.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:48.021 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:47.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:48.021 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:47.631+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:48.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:48.254+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:48.254+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:48.254+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:48.254+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:48.019+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:48.019+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:48.019+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:48.020+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:48.988 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:48.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:48.989 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:48.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:48.989 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:48.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:48.989 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:48.608+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:48.989 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:48.987+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:48.989 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:48.987+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:48.989 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:48.987+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:49.225+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:49.225+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:49.225+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:49.226+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:48.988+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:50.004 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:49.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:50.004 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:49.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:50.005 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:49.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:50.005 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:49.626+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:50.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:50.202+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:50.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:50.202+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:50.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:50.202+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:50.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:50.202+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:50.273 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:50.003+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:50.274 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:50.003+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:50.274 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:50.003+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:50.274 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:50.004+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:51.011 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:50.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:51.011 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:50.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:51.011 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:50.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:51.011 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:50.644+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:51.203+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:51.203+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:51.203+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:51.204+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:51.009+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:51.009+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:51.009+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:51.010+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:51.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.726+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:46:51.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.726+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:46:51.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.726+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f17e8020ff0 tx=0x7f17e802c000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:46:51.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.726+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f17e8020ff0 tx=0x7f17e802c000 comp rx=0 tx=0).stop 2024-09-10T03:46:51.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.727+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:46:51.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.727+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:51.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.727+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:51.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.727+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:51.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.727+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d805a5e0 2024-09-10T03:46:51.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.727+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:46:51.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.727+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:46:51.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.727+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:46:51.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.727+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:51.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.727+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048400 con 0x7f17d805a5e0 2024-09-10T03:46:51.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.728+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f17d800b2f0 tx=0x7f17e8014590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:46:51.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.728+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055050 con 0x7f17d805a5e0 2024-09-10T03:46:51.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.729+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d805a5e0 2024-09-10T03:46:51.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:51.730+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d805a5e0 2024-09-10T03:46:52.051 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:51.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:52.051 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:51.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:52.051 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:51.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:52.051 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:51.693+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:52.051 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:46:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:46:51.727+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 451 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:46:52.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:52.160+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:52.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:52.160+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:52.160+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:52.161+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:52.048+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:52.048+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:52.048+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:52.049+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:53.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:52.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:53.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:52.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:53.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:52.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:53.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:52.704+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:53.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:53.342+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:46:53.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:53.342+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:46:53.345 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:53.164+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:53.345 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:53.164+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:53.345 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:53.164+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:53.345 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:53.165+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:53.346 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:53.082+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:53.346 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:53.082+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:53.346 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:53.082+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:53.346 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:53.083+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:54.077 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:53.675+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:54.077 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:53.675+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:54.077 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:53.675+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:54.077 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:53.676+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:54.200+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:54.200+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:54.200+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:54.201+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:54.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:54.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:54.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:54.076+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:55.064 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:54.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:55.064 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:54.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:55.064 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:54.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:55.064 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:54.661+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:55.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:55.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:55.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:55.177+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:55.062+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:55.062+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:55.062+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:55.063+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:56.062 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:55.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:56.062 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:55.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:56.062 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:55.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:56.062 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:55.703+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:56.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:56.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:56.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:56.060+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:56.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:56.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:56.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:56.163+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:56.729 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.728+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:46:56.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.728+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:46:56.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.728+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f17d800b2f0 tx=0x7f17e8014590 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:46:56.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.728+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f17d800b2f0 tx=0x7f17e8014590 comp rx=0 tx=0).stop 2024-09-10T03:46:56.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.728+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:46:56.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.728+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:56.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.728+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:56.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.728+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:46:56.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.728+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d8047ae0 2024-09-10T03:46:56.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.728+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:46:56.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.728+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:46:56.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.728+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:46:56.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.728+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:46:56.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.728+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80485d0 con 0x7f17d8047ae0 2024-09-10T03:46:56.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.728+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f17e803c780 tx=0x7f17e80096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:46:56.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.729+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e800d920 con 0x7f17d8047ae0 2024-09-10T03:46:56.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.729+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d8047ae0 2024-09-10T03:46:56.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:46:56.730+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d8047ae0 2024-09-10T03:46:57.059 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:46:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:46:56.728+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 468 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:46:57.060 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:56.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:57.060 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:56.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:57.060 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:56.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:57.060 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:56.736+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:57.211+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:57.211+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:57.211+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:57.212+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:57.058+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:57.058+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:57.058+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:57.058+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:58.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:57.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:58.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:57.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:58.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:57.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:58.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:57.699+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:58.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:58.167+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:58.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:58.167+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:58.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:58.167+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:58.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:58.168+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:58.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:58.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:58.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:58.081+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:59.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:58.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:59.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:58.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:59.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:58.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:46:59.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:58.734+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:46:59.490 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:59.199+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:59.490 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:59.199+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:59.490 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:59.199+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:46:59.490 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:46:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:46:59.200+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:46:59.490 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:59.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:59.490 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:59.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:59.491 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:59.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:46:59.491 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:46:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:46:59.118+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:00.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:59.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:00.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:59.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:00.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:59.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:46:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:46:59.696+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:00.389 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:00.220+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:00.389 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:00.220+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:00.389 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:00.220+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:00.389 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:00.221+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:00.389 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:00.118+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:00.389 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:00.118+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:00.389 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:00.118+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:00.390 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:00.118+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:01.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:00.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:01.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:00.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:01.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:00.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:01.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:00.698+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:01.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:01.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:01.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:01.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:01.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:01.258+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:01.092+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:01.092+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:01.092+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:01.093+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:01.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.728+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:47:01.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.728+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:47:01.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.728+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f17e803c780 tx=0x7f17e80096d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:47:01.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.729+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f17e803c780 tx=0x7f17e80096d0 comp rx=0 tx=0).stop 2024-09-10T03:47:01.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.729+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:47:01.730 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.729+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.729+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.729+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.729+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d805a5e0 2024-09-10T03:47:01.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.729+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:47:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.730+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:47:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.730+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:47:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.730+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.730+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80487a0 con 0x7f17d805a5e0 2024-09-10T03:47:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.730+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f17e8003da0 tx=0x7f17e800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:47:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.731+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0061070 con 0x7f17d805a5e0 2024-09-10T03:47:01.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.731+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0047d80 con 0x7f17d805a5e0 2024-09-10T03:47:01.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:01.732+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d805a5e0 2024-09-10T03:47:02.065 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:01.724+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:02.065 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:01.724+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:02.065 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:01.724+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:02.065 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:01.725+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:02.065 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:47:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:47:01.729+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 485 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:47:02.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:02.251+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:02.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:02.251+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:02.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:02.251+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:02.252+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:02.063+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:02.063+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:02.063+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:02.064+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:03.060 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:02.728+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:03.060 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:02.728+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:03.061 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:02.728+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:03.061 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:02.729+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:03.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:03.282+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:03.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:03.282+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:03.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:03.282+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:03.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:03.283+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:03.059+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:03.059+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:03.059+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:03.060+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:03.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:03.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:03.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:03.712+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:04.297 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:04.022+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:04.298 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:04.022+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:04.298 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:04.022+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:04.298 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:04.023+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:04.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:04.296+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:04.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:04.296+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:04.296+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:04.297+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:04.990 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:04.690+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:04.990 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:04.690+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:04.990 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:04.690+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:04.990 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:04.690+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:05.285 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:04.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:05.285 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:04.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:05.285 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:04.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:05.285 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:04.990+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:05.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:05.283+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:05.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:05.283+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:05.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:05.283+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:05.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:05.284+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:06.009 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:05.733+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:06.010 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:05.733+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:06.010 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:05.733+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:06.010 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:05.733+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:06.328 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:06.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:06.328 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:06.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:06.328 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:06.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:06.328 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:06.009+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:06.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:06.327+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:06.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:06.327+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:06.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:06.327+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:06.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:06.328+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:06.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.729+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:47:06.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.729+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:47:06.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.729+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f17e8003da0 tx=0x7f17e800f000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:47:06.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.730+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f17e8003da0 tx=0x7f17e800f000 comp rx=0 tx=0).stop 2024-09-10T03:47:06.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.730+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:47:06.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.730+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:06.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.730+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:06.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.730+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:06.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.730+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d8047ae0 2024-09-10T03:47:06.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.730+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:47:06.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.730+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:47:06.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.730+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:47:06.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.730+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:06.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.730+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048970 con 0x7f17d8047ae0 2024-09-10T03:47:06.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.731+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f17e003a3c0 tx=0x7f17e001a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:47:06.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.732+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028070 con 0x7f17d8047ae0 2024-09-10T03:47:06.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.732+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e005ae50 con 0x7f17d8047ae0 2024-09-10T03:47:06.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:06.733+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063030 con 0x7f17d8047ae0 2024-09-10T03:47:07.057 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:47:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:47:06.729+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 502 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:47:07.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:06.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:07.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:06.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:07.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:06.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:07.058 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:06.698+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:07.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:07.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:07.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:07.056+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:07.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:07.056+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:07.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:07.056+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:07.651 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:07.357+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:07.651 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:07.357+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:07.651 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:07.357+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:07.651 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:07.358+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:08.056 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:07.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:08.056 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:07.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:08.056 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:07.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:08.056 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:07.649+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:08.054+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:08.054+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:08.054+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:08.055+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:08.642 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:08.383+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:08.642 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:08.383+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:08.642 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:08.383+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:08.642 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:08.383+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:08.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:08.703+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:47:08.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:08.703+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:47:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:08.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:08.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:08.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:08.642+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:09.344 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:09.088+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:09.344 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:09.088+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:09.344 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:09.088+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:09.344 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:09.089+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:09.674 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:09.342+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:09.674 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:09.342+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:09.674 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:09.342+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:09.674 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:09.343+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:10.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:09.673+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:10.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:09.673+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:10.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:09.673+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:10.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:09.674+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:10.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:10.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:10.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:10.065+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:10.684 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:10.352+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:10.684 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:10.352+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:10.684 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:10.352+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:10.684 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:10.352+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:11.078 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:10.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:11.078 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:10.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:11.078 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:10.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:11.078 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:10.683+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:11.076+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:11.076+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:11.076+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:11.077+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:11.723 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:11.388+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:11.723 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:11.388+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:11.723 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:11.388+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:11.723 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:11.389+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:11.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.730+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:47:11.731 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.730+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:47:11.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.730+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f17e003a3c0 tx=0x7f17e001a8c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:47:11.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.730+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f17e003a3c0 tx=0x7f17e001a8c0 comp rx=0 tx=0).stop 2024-09-10T03:47:11.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.730+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:47:11.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.730+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:11.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.730+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:11.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.730+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:11.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.730+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d805a5e0 2024-09-10T03:47:11.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.730+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:47:11.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.730+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:47:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.731+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:47:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.731+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.731+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048970 con 0x7f17d805a5e0 2024-09-10T03:47:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.732+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f17e003ee10 tx=0x7f17e0050fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:47:11.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.734+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0015d70 con 0x7f17d805a5e0 2024-09-10T03:47:11.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.735+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e001ee70 con 0x7f17d805a5e0 2024-09-10T03:47:11.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:11.736+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063030 con 0x7f17d805a5e0 2024-09-10T03:47:12.054 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:11.721+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:12.054 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:11.721+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:12.054 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:11.721+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:12.054 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:11.722+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:12.055 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:47:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:47:11.730+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 519 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:47:12.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:12.053+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:12.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:12.053+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:12.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:12.053+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:12.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:12.054+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:12.686 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:12.358+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:12.687 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:12.358+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:12.687 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:12.358+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:12.687 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:12.359+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:13.035 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:12.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:13.035 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:12.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:13.035 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:12.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:13.036 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:12.686+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:13.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:13.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:13.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:13.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:13.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:13.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:13.035+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:13.716 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:13.370+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:13.717 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:13.370+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:13.717 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:13.370+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:13.717 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:13.371+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:14.076 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:13.715+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:14.077 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:13.715+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:14.077 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:13.715+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:14.077 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:13.716+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:14.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:14.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:14.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:14.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:14.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:14.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:14.076+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:14.598 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:14.329+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:14.598 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:14.329+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:14.598 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:14.329+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:14.598 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:14.330+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:14.849 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:14.743+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:14.850 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:14.743+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:14.850 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:14.743+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:14.850 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:14.744+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:15.068+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:15.068+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:15.068+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:15.069+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:15.695 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:15.374+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:15.695 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:15.374+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:15.695 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:15.374+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:15.695 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:15.375+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:16.068 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:15.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:16.068 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:15.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:16.068 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:15.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:16.068 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:15.760+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:16.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:16.066+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:16.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:16.066+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:16.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:16.066+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:16.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:16.067+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:16.723 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:16.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:16.724 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:16.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:16.724 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:16.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:16.724 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:16.390+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:16.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.730+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:47:16.732 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.730+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:47:16.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.730+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f17e003ee10 tx=0x7f17e0050fc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:47:16.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.730+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f17e003ee10 tx=0x7f17e0050fc0 comp rx=0 tx=0).stop 2024-09-10T03:47:16.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.731+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:47:16.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.731+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:16.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.731+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:16.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.731+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:16.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.731+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d8047ae0 2024-09-10T03:47:16.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.731+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:47:16.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.731+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:47:16.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.732+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:47:16.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.732+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:16.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.732+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048970 con 0x7f17d8047ae0 2024-09-10T03:47:16.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.732+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f17e8044f10 tx=0x7f17e0011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:47:16.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.732+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8035270 con 0x7f17d8047ae0 2024-09-10T03:47:16.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.733+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e80093a0 con 0x7f17d8047ae0 2024-09-10T03:47:16.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:16.734+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d8047ae0 2024-09-10T03:47:17.046 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:47:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:47:16.730+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 536 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:47:17.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:16.722+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:17.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:16.722+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:17.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:16.722+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:17.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:16.723+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:17.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:17.044+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:17.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:17.044+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:17.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:17.044+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:17.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:17.045+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:17.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:17.357+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:17.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:17.357+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:17.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:17.357+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:17.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:17.358+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:18.042 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:17.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:18.042 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:17.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:18.042 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:17.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:18.042 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:17.696+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:18.040+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:18.040+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:18.040+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:18.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:18.041+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:18.705 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:18.372+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:18.705 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:18.372+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:18.705 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:18.372+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:18.705 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:18.372+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:19.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:18.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:19.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:18.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:19.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:18.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:19.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:18.704+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:19.419 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:19.090+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:19.420 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:19.090+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:19.420 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:19.090+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:19.420 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:19.091+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:19.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:19.418+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:19.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:19.418+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:19.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:19.418+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:19.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:19.419+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:20.058 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:19.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:20.058 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:19.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:20.058 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:19.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:20.058 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:19.736+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:20.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:20.056+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:20.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:20.056+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:20.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:20.056+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:20.057+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:20.688 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:20.436+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:20.688 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:20.436+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:20.689 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:20.436+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:20.689 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:20.437+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:20.689 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:20.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:20.689 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:20.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:20.689 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:20.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:20.689 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:20.687+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:21.035+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:21.035+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:21.035+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:21.035+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:21.681 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:21.399+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:21.681 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:21.399+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:21.681 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:21.399+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:21.681 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:21.400+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:21.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.731+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:47:21.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.731+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:47:21.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.731+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f17e8044f10 tx=0x7f17e0011e80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:47:21.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.731+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f17e8044f10 tx=0x7f17e0011e80 comp rx=0 tx=0).stop 2024-09-10T03:47:21.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.731+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:47:21.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.731+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:21.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.731+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:21.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.731+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:21.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.731+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d805a5e0 2024-09-10T03:47:21.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.731+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:47:21.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.732+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:47:21.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.732+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:47:21.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.732+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:21.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.732+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048970 con 0x7f17d805a5e0 2024-09-10T03:47:21.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.733+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f17d8011cf0 tx=0x7f17e802e140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:47:21.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.733+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c070 con 0x7f17d805a5e0 2024-09-10T03:47:21.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.734+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e005ae50 con 0x7f17d805a5e0 2024-09-10T03:47:21.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:21.734+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d805a5e0 2024-09-10T03:47:22.056 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:47:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:47:21.731+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 553 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:47:22.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:21.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:22.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:21.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:22.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:21.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:22.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:21.681+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:22.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:22.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:22.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:22.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:22.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:22.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:22.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:22.056+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:22.654 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:22.390+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:22.654 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:22.390+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:22.654 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:22.390+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:22.654 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:22.391+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:23.056 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:22.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:23.056 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:22.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:23.056 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:22.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:23.056 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:22.653+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:23.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:23.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:23.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:23.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:23.055+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:23.390+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:23.390+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:23.390+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:23.391+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:23.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:23.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:23.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:23.629+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:24.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:24.063+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:47:24.064 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:24.063+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:47:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:24.085+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:24.085+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:24.085+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:24.086+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:24.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:24.436+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:24.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:24.436+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:24.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:24.436+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:24.437+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:24.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:24.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:24.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:24.615+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:25.440 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:25.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:25.441 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:25.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:25.441 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:25.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:25.441 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:25.117+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:25.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:25.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:25.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:25.440+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:25.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:25.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:25.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:25.647+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:26.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:26.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:26.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:26.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:26.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:26.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:26.117+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:26.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.731+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:47:26.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.731+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:47:26.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.731+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f17d8011cf0 tx=0x7f17e802e140 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:47:26.733 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:26.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:26.733 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:26.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:26.733 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:26.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:26.734 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:26.476+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:26.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.731+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f17d8011cf0 tx=0x7f17e802e140 comp rx=0 tx=0).stop 2024-09-10T03:47:26.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.732+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:47:26.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.732+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:26.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.732+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:26.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.732+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:26.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.732+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d8047ae0 2024-09-10T03:47:26.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.732+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:47:26.735 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:26.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:26.735 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:26.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:26.735 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:26.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:26.735 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:26.686+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:26.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.732+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:47:26.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.733+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:47:26.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.733+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:26.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.733+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048970 con 0x7f17d8047ae0 2024-09-10T03:47:26.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.733+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f17e8035cb0 tx=0x7f17e0015710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:47:26.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.734+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c040 con 0x7f17d8047ae0 2024-09-10T03:47:26.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.734+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0018310 con 0x7f17d8047ae0 2024-09-10T03:47:26.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:26.735+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0014170 con 0x7f17d8047ae0 2024-09-10T03:47:27.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:47:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:47:26.731+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 570 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:47:27.470 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:27.081+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:27.470 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:27.081+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:27.470 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:27.081+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:27.470 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:27.081+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:27.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:27.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:27.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:27.649+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:27.468+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:27.468+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:27.468+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:27.469+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:28.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:28.071+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:28.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:28.071+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:28.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:28.071+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:28.071+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:28.448+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:28.448+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:28.448+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:28.449+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:28.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:28.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:28.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:28.610+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:29.448 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:29.097+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:29.448 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:29.097+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:29.448 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:29.097+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:29.448 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:29.098+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:29.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:29.446+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:29.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:29.446+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:29.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:29.446+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:29.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:29.446+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:29.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:29.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:29.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:29.631+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:30.407 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:30.088+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:30.407 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:30.088+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:30.407 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:30.088+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:30.407 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:30.088+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:30.680 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:30.404+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:30.680 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:30.404+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:30.680 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:30.404+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:30.680 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:30.405+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:31.026 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:30.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:31.026 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:30.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:31.026 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:30.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:31.026 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:30.678+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:31.081+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:31.081+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:31.081+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:31.081+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:31.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.731+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:47:31.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.731+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:47:31.733 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.731+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f17e8035cb0 tx=0x7f17e0015710 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:47:31.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.731+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f17e8035cb0 tx=0x7f17e0015710 comp rx=0 tx=0).stop 2024-09-10T03:47:31.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.731+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:47:31.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.731+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:31.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.731+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:31.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.731+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:31.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.731+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d805a5e0 2024-09-10T03:47:31.734 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.731+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:47:31.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:31.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:31.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:31.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:31.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:31.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:31.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:31.639+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:31.735 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:31.419+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:31.735 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:31.419+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:31.735 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:31.419+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:31.735 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:31.420+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:31.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.731+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:47:31.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.733+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:47:31.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.733+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:31.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.733+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048970 con 0x7f17d805a5e0 2024-09-10T03:47:31.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.734+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f17e800cf90 tx=0x7f17e0009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:47:31.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.735+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055040 con 0x7f17d805a5e0 2024-09-10T03:47:31.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.736+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e80093a0 con 0x7f17d805a5e0 2024-09-10T03:47:31.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:31.736+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e800d750 con 0x7f17d805a5e0 2024-09-10T03:47:32.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:47:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:47:31.731+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 591 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:47:32.441 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:32.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:32.441 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:32.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:32.441 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:32.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:32.441 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:32.079+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:32.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:32.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:32.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:32.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:32.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:32.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:32.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:32.439+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:32.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:32.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:32.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:32.686+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:33.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:33.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:33.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:33.064+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:33.477+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:33.477+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:33.477+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:33.477+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:33.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:33.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:33.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:33.677+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:34.047+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:34.047+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:34.047+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:34.048+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:34.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:34.471+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:34.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:34.471+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:34.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:34.471+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:34.472+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:34.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:34.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:34.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:34.638+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:35.472 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:35.087+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:35.472 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:35.087+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:35.472 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:35.087+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:35.472 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:35.088+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:35.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:35.470+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:35.470+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:35.470+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:35.471+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:35.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:35.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:35.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:35.638+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:36.073+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:36.073+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:36.073+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:36.074+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:36.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:36.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:36.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:36.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:36.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:36.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:36.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:36.610+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:36.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.734+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:47:36.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.734+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:47:36.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.734+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f17e800cf90 tx=0x7f17e0009190 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:47:36.735 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:36.502+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:36.735 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:36.502+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:36.735 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:36.502+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:36.735 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:36.503+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:36.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.734+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f17e800cf90 tx=0x7f17e0009190 comp rx=0 tx=0).stop 2024-09-10T03:47:36.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.734+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:47:36.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.734+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:36.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.734+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:36.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.734+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:36.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.734+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d8047ae0 2024-09-10T03:47:36.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.734+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:47:36.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.734+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:47:36.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.734+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:47:36.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.734+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:36.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.734+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80485d0 con 0x7f17d8047ae0 2024-09-10T03:47:36.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.735+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f17d802cbb0 tx=0x7f17e801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:47:36.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.736+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055040 con 0x7f17d8047ae0 2024-09-10T03:47:36.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.736+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d8047ae0 2024-09-10T03:47:36.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:36.737+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e800dc40 con 0x7f17d8047ae0 2024-09-10T03:47:37.049 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:47:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:47:36.732+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 624 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:47:37.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:37.047+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:37.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:37.047+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:37.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:37.047+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:37.048+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:37.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:37.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:37.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:37.488+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:37.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:37.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:37.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:37.649+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:38.033+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:38.033+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:38.033+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:38.033+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:38.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:38.463+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:38.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:38.463+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:38.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:38.463+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:38.464+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:38.623+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:38.623+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:38.623+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:38.624+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:38.998+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:38.998+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:38.998+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:38.999+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:39.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:39.488+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:47:39.489 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:39.488+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:47:39.631 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:39.485+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:39.631 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:39.485+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:39.631 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:39.485+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:39.631 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:39.486+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:39.965 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:39.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:39.965 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:39.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:39.965 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:39.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:39.965 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:39.630+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:39.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:39.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:39.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:39.964+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:40.481+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:40.481+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:40.481+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:40.481+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:40.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:40.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:40.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:40.631+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:41.235 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:40.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:41.235 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:40.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:41.235 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:40.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:41.235 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:40.971+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:41.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.733+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:47:41.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.733+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:47:41.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.733+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f17d802cbb0 tx=0x7f17e801e000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:47:41.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.733+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f17d802cbb0 tx=0x7f17e801e000 comp rx=0 tx=0).stop 2024-09-10T03:47:41.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.733+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:47:41.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.733+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:41.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.733+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:41.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.733+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:41.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.733+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d805a5e0 2024-09-10T03:47:41.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.733+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:47:41.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.734+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:47:41.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.734+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:47:41.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.734+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:41.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.734+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80485d0 con 0x7f17d805a5e0 2024-09-10T03:47:41.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.734+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f17e0011ec0 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:47:41.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.735+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0006be0 con 0x7f17d805a5e0 2024-09-10T03:47:41.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.735+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0047370 con 0x7f17d805a5e0 2024-09-10T03:47:41.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:41.736+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0014170 con 0x7f17d805a5e0 2024-09-10T03:47:41.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:41.490+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:41.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:41.490+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:41.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:41.490+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:41.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:41.491+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:41.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:41.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:41.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:41.658+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:41.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:47:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:47:41.733+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 658 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:47:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:41.984+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:41.984+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:41.984+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:41.985+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:42.537+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:42.537+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:42.537+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:42.537+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:42.691+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:42.691+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:42.691+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:42.691+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:43.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:42.937+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:43.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:42.937+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:43.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:42.937+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:42.938+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:43.506+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:43.506+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:43.506+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:43.507+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:43.722+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:43.722+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:43.722+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:43.723+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:43.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:43.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:43.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:43.920+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:44.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:44.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:44.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:44.476+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:44.719+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:44.719+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:44.719+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:44.720+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:45.334 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:44.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:45.334 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:44.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:45.334 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:44.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:45.334 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:44.906+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:45.633 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:45.495+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:45.633 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:45.495+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:45.634 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:45.495+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:45.634 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:45.495+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:45.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:45.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:45.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:45.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:45.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:45.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:45.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:45.686+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:46.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:45.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:46.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:45.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:46.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:45.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:46.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:45.955+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:46.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:46.535+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:46.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:46.535+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:46.698 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:46.535+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:46.698 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:46.536+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:46.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.733+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:47:46.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.733+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:47:46.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.733+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f17e0011ec0 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:47:46.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.734+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f17e0011ec0 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:47:46.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.734+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:47:46.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.734+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:46.735 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.734+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:46.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.734+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:46.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.734+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80485d0 con 0x7f17d8047ae0 2024-09-10T03:47:46.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.734+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:47:46.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.734+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:47:46.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.735+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:47:46.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.735+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:46.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.735+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048970 con 0x7f17d8047ae0 2024-09-10T03:47:46.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.735+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f17e8040fa0 tx=0x7f17e0061000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:47:46.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.736+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055070 con 0x7f17d8047ae0 2024-09-10T03:47:46.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.737+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d8047ae0 2024-09-10T03:47:46.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:46.738+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8014d10 con 0x7f17d8047ae0 2024-09-10T03:47:47.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:47:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:47:46.734+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 690 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:47:47.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:46.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:46.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:46.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:46.919+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:46.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:46.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:46.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:46.697+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:47.517+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:47.517+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:47.517+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:47.517+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:47.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:47.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:47.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:47.667+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:48.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:47.932+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:48.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:47.932+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:48.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:47.932+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:48.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:47.933+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:48.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:48.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:48.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:48.477+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:48.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:48.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:48.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:48.620+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:49.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:48.936+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:49.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:48.936+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:49.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:48.936+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:49.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:48.936+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:49.464+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:49.464+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:49.464+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:49.464+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:49.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:49.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:49.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:49.657+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:49.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:49.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:49.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:49.921+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:50.428+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:50.428+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:50.428+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:50.429+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:50.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:50.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:50.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:50.632+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:50.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:50.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:50.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:50.904+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:51.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.734+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:47:51.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.734+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:47:51.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.734+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f17e8040fa0 tx=0x7f17e0061000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:47:51.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.734+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f17e8040fa0 tx=0x7f17e0061000 comp rx=0 tx=0).stop 2024-09-10T03:47:51.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.734+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:47:51.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.734+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:51.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.734+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:51.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.734+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:51.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.734+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d805a5e0 2024-09-10T03:47:51.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.734+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:47:51.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:51.458+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:51.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:51.458+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:51.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:51.458+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:51.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:51.458+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:51.737 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:51.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:51.738 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:51.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:51.738 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:51.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:51.738 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:51.646+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:51.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.735+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:47:51.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.736+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:47:51.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.736+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:51.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.736+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048b40 con 0x7f17d805a5e0 2024-09-10T03:47:51.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.736+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f17e8055040 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:47:51.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.737+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8026390 con 0x7f17d805a5e0 2024-09-10T03:47:51.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.737+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e802ed00 con 0x7f17d805a5e0 2024-09-10T03:47:51.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:51.738+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8014d10 con 0x7f17d805a5e0 2024-09-10T03:47:52.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:47:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:47:51.734+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 722 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:47:52.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:51.913+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:52.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:51.913+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:52.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:51.913+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:51.914+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:52.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:52.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:52.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:52.451+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:52.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:52.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:52.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:52.606+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:52.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:52.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:52.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:52.955+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:53.470+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:53.470+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:53.470+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:53.471+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:53.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:53.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:53.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:53.644+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:54.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:54.003+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:54.003+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:54.003+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:54.498+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:54.498+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:54.498+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:54.498+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:54.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:54.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:54.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:54.623+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:54.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:54.847+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:47:54.848 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:54.847+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:47:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:55.007+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:55.007+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:55.007+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:55.008+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:55.450+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:55.450+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:55.450+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:55.451+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:55.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:55.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:55.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:55.661+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:56.014+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:56.014+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:56.014+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:56.015+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:56.736 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.735+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:47:56.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.735+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:47:56.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.735+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f17e8055040 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:47:56.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.735+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f17e8055040 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:47:56.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.735+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:47:56.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.735+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:56.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.735+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:56.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.735+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:47:56.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.735+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d8047ae0 2024-09-10T03:47:56.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.735+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:47:56.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.735+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:47:56.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.735+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:47:56.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.735+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:47:56.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.735+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048970 con 0x7f17d8047ae0 2024-09-10T03:47:56.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.736+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f17d8018b80 tx=0x7f17e8005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:47:56.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.737+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c030 con 0x7f17d8047ae0 2024-09-10T03:47:56.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.737+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0015390 con 0x7f17d8047ae0 2024-09-10T03:47:56.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:47:56.738+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0014170 con 0x7f17d8047ae0 2024-09-10T03:47:56.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:47:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:47:56.735+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 754 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:47:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:56.501+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:56.501+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:56.501+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:56.501+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:56.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:56.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:56.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:56.663+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:57.025+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:57.025+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:57.025+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:57.025+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:57.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:57.482+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:57.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:57.482+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:57.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:57.482+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:57.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:57.482+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:57.667+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:57.667+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:57.667+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:57.667+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:57.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:57.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:57.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:57.992+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:58.715 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:58.445+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:58.715 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:58.445+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:58.715 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:58.445+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:58.715 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:58.446+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:58.990 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:58.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:58.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:58.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:58.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:58.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:47:58.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:58.715+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:47:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:58.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:58.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:58.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:47:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:58.990+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:47:59.734 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:59.458+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:59.734 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:59.458+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:59.734 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:59.458+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:47:59.734 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:47:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:47:59.459+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:00.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:59.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:00.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:59.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:00.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:59.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:00.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:47:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:47:59.961+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:59.732+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:59.732+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:59.732+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:47:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:47:59.733+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:00.778 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:00.433+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:00.778 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:00.433+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:00.778 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:00.433+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:00.778 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:00.434+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:01.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:00.776+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:01.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:00.776+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:01.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:00.776+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:01.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:00.777+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:00.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:00.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:00.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:00.990+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:01.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:01.423+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:01.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:01.423+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:01.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:01.423+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:01.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:01.424+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:01.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.735+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:48:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.735+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:48:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.735+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f17d8018b80 tx=0x7f17e8005df0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:48:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.735+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f17d8018b80 tx=0x7f17e8005df0 comp rx=0 tx=0).stop 2024-09-10T03:48:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.735+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:48:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.735+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.735+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.735+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.735+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80485d0 con 0x7f17d805a5e0 2024-09-10T03:48:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.735+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:48:01.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.736+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:48:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.737+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:48:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.737+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.737+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80487a0 con 0x7f17d805a5e0 2024-09-10T03:48:01.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.737+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f17e800ffd0 tx=0x7f17e0028000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:48:01.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.739+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055040 con 0x7f17d805a5e0 2024-09-10T03:48:01.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.739+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d805a5e0 2024-09-10T03:48:01.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:01.741+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e800ccf0 con 0x7f17d805a5e0 2024-09-10T03:48:02.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:48:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:48:01.735+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 796 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:48:02.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:01.774+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:01.774+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:01.774+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:01.775+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:01.972+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:01.972+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:01.972+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:01.973+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:02.760 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:02.433+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:02.760 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:02.433+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:02.760 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:02.433+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:02.761 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:02.434+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:03.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:02.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:03.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:02.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:02.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:02.760+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:02.935+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:02.935+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:02.935+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:02.935+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:03.800 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:03.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:03.800 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:03.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:03.800 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:03.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:03.801 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:03.440+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:04.064 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:03.799+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:04.064 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:03.799+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:04.064 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:03.799+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:04.064 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:03.800+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:04.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:03.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:04.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:03.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:04.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:03.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:04.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:03.919+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:04.414+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:04.414+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:04.414+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:04.415+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:05.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:04.847+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:05.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:04.847+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:04.847+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:04.848+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:04.962+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:04.962+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:04.962+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:04.962+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:05.399+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:05.399+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:05.399+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:05.400+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:06.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:05.876+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:06.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:05.876+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:06.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:05.876+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:06.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:05.876+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:05.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:05.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:05.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:05.943+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:06.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.735+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:48:06.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.735+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:48:06.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.735+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f17e800ffd0 tx=0x7f17e0028000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:48:06.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.735+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f17e800ffd0 tx=0x7f17e0028000 comp rx=0 tx=0).stop 2024-09-10T03:48:06.737 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.735+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:48:06.738 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:06.403+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:06.738 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:06.403+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:06.738 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:06.403+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:06.738 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:06.404+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:06.738 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.735+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:06.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.735+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:06.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.735+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:06.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.735+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80485d0 con 0x7f17d8047ae0 2024-09-10T03:48:06.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.735+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:48:06.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.736+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:48:06.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.736+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:48:06.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.736+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:06.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.736+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048970 con 0x7f17d8047ae0 2024-09-10T03:48:06.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.736+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f17e0051000 tx=0x7f17e800c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:48:06.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.738+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0061070 con 0x7f17d8047ae0 2024-09-10T03:48:06.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.738+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e00228a0 con 0x7f17d8047ae0 2024-09-10T03:48:06.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:06.739+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e003c210 con 0x7f17d8047ae0 2024-09-10T03:48:06.989 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:06.878+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:06.989 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:06.878+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:06.990 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:06.878+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:06.990 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:06.879+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:06.990 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:48:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:48:06.736+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 859 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:48:07.265 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:06.987+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:07.265 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:06.987+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:07.265 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:06.987+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:07.265 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:06.988+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:07.445+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:07.445+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:07.445+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:07.446+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:08.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:07.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:08.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:07.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:08.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:07.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:07.863+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:08.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:08.037+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:08.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:08.037+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:08.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:08.037+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:08.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:08.038+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:08.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:08.466+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:08.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:08.466+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:08.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:08.466+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:08.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:08.467+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:09.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:08.856+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:08.856+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:08.856+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:08.857+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:09.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:09.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:09.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:09.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:09.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:09.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:09.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:09.061+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:09.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:09.475+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:09.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:09.475+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:09.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:09.475+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:09.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:09.476+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:10.104+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:10.104+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:10.104+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:10.105+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:09.892+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:09.892+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:09.892+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:09.893+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:10.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:10.334+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:48:10.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:10.334+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:48:10.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:10.430+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:10.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:10.430+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:10.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:10.430+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:10.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:10.431+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:11.068+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:11.068+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:11.068+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:11.069+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:10.937+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:10.937+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:10.937+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:10.937+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:11.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.737+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:48:11.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.737+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:48:11.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.737+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f17e0051000 tx=0x7f17e800c710 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:48:11.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.737+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f17e0051000 tx=0x7f17e800c710 comp rx=0 tx=0).stop 2024-09-10T03:48:11.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.737+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:48:11.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.737+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:11.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.737+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:11.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.737+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:11.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.737+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80485d0 con 0x7f17d805a5e0 2024-09-10T03:48:11.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.737+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:48:11.740 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:11.424+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:11.740 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:11.424+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:11.740 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:11.425+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:11.740 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:11.425+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:11.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.738+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:48:11.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.738+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:48:11.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.738+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:11.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.739+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048b40 con 0x7f17d805a5e0 2024-09-10T03:48:11.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.739+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f17e002dc80 tx=0x7f17e003c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:48:11.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.739+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0061070 con 0x7f17d805a5e0 2024-09-10T03:48:11.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.739+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e001ee70 con 0x7f17d805a5e0 2024-09-10T03:48:11.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:11.740+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d805a5e0 2024-09-10T03:48:12.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:48:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:48:11.737+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 920 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:48:12.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:11.917+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:11.917+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:11.917+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:11.918+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:12.475 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:12.089+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:12.475 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:12.089+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:12.475 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:12.089+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:12.475 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:12.089+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:12.473+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:12.473+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:12.473+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:12.474+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:13.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:13.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:13.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:13.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:13.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:13.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:13.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:13.117+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:13.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:12.923+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:13.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:12.923+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:13.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:12.923+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:13.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:12.923+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:13.505+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:13.505+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:13.505+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:13.506+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:14.162 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:13.892+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:14.162 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:13.892+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:14.163 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:13.892+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:14.163 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:13.893+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:14.529 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:14.161+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:14.529 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:14.161+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:14.529 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:14.161+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:14.529 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:14.162+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:14.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:14.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:14.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:14.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:14.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:14.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:14.528+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:15.114+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:15.114+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:15.114+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:15.115+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:14.880+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:14.880+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:14.880+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:14.881+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:15.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:15.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:15.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:15.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:15.569+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:16.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:16.084+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:16.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:16.084+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:16.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:16.084+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:16.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:16.085+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:15.914+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:15.914+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:15.914+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:16.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:15.915+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:16.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.737+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:48:16.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.737+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:48:16.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.738+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f17e002dc80 tx=0x7f17e003c550 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:48:16.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.738+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f17e002dc80 tx=0x7f17e003c550 comp rx=0 tx=0).stop 2024-09-10T03:48:16.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.738+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:48:16.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.738+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:16.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.738+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:16.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.738+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:16.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.738+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d8047ae0 2024-09-10T03:48:16.740 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.738+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:48:16.741 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:16.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:16.741 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:16.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:16.741 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:16.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:16.741 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:16.568+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:16.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.738+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:48:16.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.739+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:48:16.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.739+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:16.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.739+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048b40 con 0x7f17d8047ae0 2024-09-10T03:48:16.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.740+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f17e8002010 tx=0x7f17e0022550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:48:16.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.742+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e800d750 con 0x7f17d8047ae0 2024-09-10T03:48:16.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.742+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d8047ae0 2024-09-10T03:48:16.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:16.742+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d8047ae0 2024-09-10T03:48:17.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:48:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:48:16.738+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 982 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:48:17.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:16.918+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:17.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:16.918+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:17.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:16.918+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:17.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:16.918+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:17.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:17.132+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:17.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:17.132+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:17.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:17.132+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:17.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:17.132+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:17.553+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:17.553+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:17.553+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:17.554+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:18.129 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:17.873+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:18.129 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:17.873+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:18.129 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:17.873+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:18.129 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:17.874+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:18.576 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:18.127+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:18.576 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:18.127+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:18.576 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:18.127+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:18.576 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:18.128+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:18.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:18.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:18.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:18.575+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:19.161 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:18.865+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:19.162 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:18.865+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:19.162 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:18.865+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:19.162 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:18.866+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:19.160+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:19.160+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:19.160+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:19.161+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:19.905 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:19.623+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:19.905 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:19.623+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:19.905 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:19.623+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:19.905 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:19.624+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:20.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:19.903+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:20.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:19.903+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:20.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:19.903+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:19.904+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:20.127+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:20.127+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:20.127+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:20.127+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:20.896 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:20.636+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:20.896 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:20.636+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:20.896 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:20.636+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:20.896 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:20.637+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:21.091+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:21.091+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:21.091+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:21.092+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:20.894+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:20.894+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:20.894+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:20.895+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:21.739 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.738+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:48:21.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.738+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:48:21.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.738+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f17e8002010 tx=0x7f17e0022550 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:48:21.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.738+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f17e8002010 tx=0x7f17e0022550 comp rx=0 tx=0).stop 2024-09-10T03:48:21.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.739+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:48:21.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.739+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:21.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.739+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:21.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.739+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:21.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.739+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d805a5e0 2024-09-10T03:48:21.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.739+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:48:21.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.740+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:48:21.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.740+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:48:21.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.740+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:21.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.740+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80487a0 con 0x7f17d805a5e0 2024-09-10T03:48:21.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.741+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f17e0031000 tx=0x7f17e8026390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:48:21.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.742+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028040 con 0x7f17d805a5e0 2024-09-10T03:48:21.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.742+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e001ee70 con 0x7f17d805a5e0 2024-09-10T03:48:21.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:21.743+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d805a5e0 2024-09-10T03:48:22.055 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:48:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:48:21.739+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 1044 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:48:22.055 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:21.847+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:22.055 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:21.848+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:22.056 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:21.848+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:22.056 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:21.848+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:22.056 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:21.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:22.056 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:21.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:22.056 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:21.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:22.056 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:21.666+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:22.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:22.054+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:22.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:22.054+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:22.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:22.054+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:22.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:22.054+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:23.049 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:22.701+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:23.049 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:22.701+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:23.049 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:22.701+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:23.049 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:22.702+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:23.050 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:22.885+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:23.050 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:22.885+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:23.050 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:22.885+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:23.050 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:22.886+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:23.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:23.048+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:23.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:23.048+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:23.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:23.048+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:23.048+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:23.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:23.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:23.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:23.738+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:24.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:23.901+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:24.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:23.901+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:24.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:23.901+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:24.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:23.902+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:24.007+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:24.007+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:24.007+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:24.008+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:24.874 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:24.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:24.874 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:24.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:24.874 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:24.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:24.874 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:24.769+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:25.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:24.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:25.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:24.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:25.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:24.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:25.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:25.000+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:25.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:24.872+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:25.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:24.872+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:25.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:24.872+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:25.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:24.873+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:25.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:25.695+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:48:25.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:25.695+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:48:26.045 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:25.754+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:26.045 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:25.754+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:26.045 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:25.754+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:26.045 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:25.755+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:26.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:25.894+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:26.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:25.894+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:26.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:25.894+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:26.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:25.895+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:26.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:26.044+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:26.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:26.044+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:26.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:26.044+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:26.045+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:26.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.739+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:48:26.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.739+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:48:26.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.739+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f17e0031000 tx=0x7f17e8026390 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:48:26.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.739+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f17e0031000 tx=0x7f17e8026390 comp rx=0 tx=0).stop 2024-09-10T03:48:26.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.739+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:48:26.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.739+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:26.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.740+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:26.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.740+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:26.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.740+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804f5b0 con 0x7f17d8047ae0 2024-09-10T03:48:26.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.740+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048750 con 0x7f17f0101a40 2024-09-10T03:48:26.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.740+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:48:26.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.740+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:48:26.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.740+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:26.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.740+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048970 con 0x7f17d8047ae0 2024-09-10T03:48:26.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.741+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f17e802b040 tx=0x7f17e0046470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:48:26.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.742+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8014e60 con 0x7f17d8047ae0 2024-09-10T03:48:26.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.742+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d8047ae0 2024-09-10T03:48:26.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:26.743+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8026390 con 0x7f17d8047ae0 2024-09-10T03:48:27.013 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:26.886+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:27.013 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:26.886+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:27.013 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:26.886+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:27.013 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:26.887+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:27.013 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:48:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:48:26.739+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 1106 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:48:27.014 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:26.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:27.014 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:26.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:27.014 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:26.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:27.014 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:26.725+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:27.011+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:27.011+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:27.011+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:27.012+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:28.029 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:27.758+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:28.030 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:27.758+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:28.030 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:27.758+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:28.030 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:27.759+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:28.030 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:27.917+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:28.030 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:27.917+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:28.030 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:27.917+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:28.030 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:27.918+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:28.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:28.028+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:28.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:28.028+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:28.028+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:28.029+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:29.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:28.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:29.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:28.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:28.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:28.771+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:28.925+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:28.925+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:28.925+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:28.926+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:28.995+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:28.995+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:29.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:28.995+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:29.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:28.996+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:30.030 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:29.931+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:30.030 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:29.931+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:30.030 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:29.931+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:30.030 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:29.932+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:30.031 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:29.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:30.031 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:29.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:30.031 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:29.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:30.031 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:29.756+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:30.031 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:30.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:30.031 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:30.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:30.031 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:30.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:30.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:30.029+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:31.006 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:30.906+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:31.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:30.906+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:31.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:30.906+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:31.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:30.907+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:31.007 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:30.716+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:31.007 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:30.716+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:31.007 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:30.716+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:31.007 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:30.717+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:31.005+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:31.005+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:31.005+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:31.006+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:31.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.740+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:48:31.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.740+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:48:31.741 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.740+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f17e802b040 tx=0x7f17e0046470 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:48:31.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.740+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f17e802b040 tx=0x7f17e0046470 comp rx=0 tx=0).stop 2024-09-10T03:48:31.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.740+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:48:31.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.740+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:31.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.740+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:31.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.740+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:31.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.740+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048750 con 0x7f17d805a5e0 2024-09-10T03:48:31.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.740+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:48:31.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.740+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:48:31.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.741+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:48:31.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.741+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:31.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.741+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048970 con 0x7f17d805a5e0 2024-09-10T03:48:31.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.742+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f17e0025ce0 tx=0x7f17e8042170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:48:31.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.742+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028040 con 0x7f17d805a5e0 2024-09-10T03:48:31.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.745+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e003aed0 con 0x7f17d805a5e0 2024-09-10T03:48:31.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:31.745+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d805a5e0 2024-09-10T03:48:32.000 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:31.923+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:32.000 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:31.923+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:32.000 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:31.923+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:32.000 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:31.924+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:32.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:31.694+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:32.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:31.694+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:32.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:31.694+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:32.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:31.695+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:32.001 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:48:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:48:31.740+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 1168 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:48:32.295 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:31.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:32.295 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:31.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:32.295 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:31.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:32.295 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:31.999+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:32.947 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:32.677+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:32.948 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:32.677+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:32.948 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:32.677+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:32.948 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:32.678+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:33.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:32.946+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:33.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:32.946+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:33.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:32.946+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:33.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:32.947+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:33.006+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:33.006+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:33.006+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:33.006+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:33.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:33.719+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:33.719+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:33.719+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:33.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:33.870+0000 7f17de7fc640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f17d4005b00 con 0x7f17d805a5e0 2024-09-10T03:48:34.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:33.961+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:34.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:33.961+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:34.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:33.961+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:34.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:33.962+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:34.003+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:34.003+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:34.003+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:34.004+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:34.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:34.870+0000 7f17de7fc640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f17d4005dc0 con 0x7f17d805a5e0 2024-09-10T03:48:34.871 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:34.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:34.872 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:34.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:34.872 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:34.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:34.872 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:34.744+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:35.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:34.983+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:35.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:34.983+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:35.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:34.983+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:35.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:34.984+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:34.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:34.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:34.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:34.966+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:35.871 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:35.870+0000 7f17de7fc640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f17d4005dc0 con 0x7f17d805a5e0 2024-09-10T03:48:36.015 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:35.740+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:36.015 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:35.740+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:36.015 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:35.740+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:36.015 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:35.741+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:36.016 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:35.970+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:36.016 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:35.970+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:36.016 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:35.970+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:36.016 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:35.970+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:36.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:36.014+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:36.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:36.014+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:36.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:36.014+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:36.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:36.015+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:36.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.740+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:48:36.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.740+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:48:36.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.740+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f17e0025ce0 tx=0x7f17e8042170 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:48:36.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.740+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f17e0025ce0 tx=0x7f17e8042170 comp rx=0 tx=0).stop 2024-09-10T03:48:36.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.740+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:48:36.742 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.740+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:36.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.740+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:36.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.740+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:36.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.740+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048750 con 0x7f17d8047ae0 2024-09-10T03:48:36.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.740+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:48:36.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.741+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:48:36.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.742+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:48:36.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.742+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:36.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.742+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048970 con 0x7f17d8047ae0 2024-09-10T03:48:36.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.742+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f17d4003470 tx=0x7f17e8000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:48:36.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.742+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0061100 con 0x7f17d8047ae0 2024-09-10T03:48:36.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.743+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e001ecc0 con 0x7f17d8047ae0 2024-09-10T03:48:36.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:36.744+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063020 con 0x7f17d8047ae0 2024-09-10T03:48:36.994 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:48:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:48:36.740+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 1230 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:48:36.994 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:36.784+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:36.995 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:36.784+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:36.995 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:36.784+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:36.995 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:36.785+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:37.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:37.062+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:37.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:37.062+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:37.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:37.062+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:37.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:37.063+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:36.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:36.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:36.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:36.992+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:37.737+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:37.737+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:37.737+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:37.738+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:37.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:37.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:37.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:37.979+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:38.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:38.079+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:38.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:38.079+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:38.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:38.079+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:38.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:38.080+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:38.705 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:38.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:38.706 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:38.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:38.706 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:38.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:38.706 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:38.703+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:39.060 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:38.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:39.060 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:38.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:39.061 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:38.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:39.061 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:38.950+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:39.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:39.059+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:39.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:39.059+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:39.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:39.059+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:39.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:39.060+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:39.932 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:39.674+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:39.932 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:39.674+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:39.933 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:39.674+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:39.933 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:39.676+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:40.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:40.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:40.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:40.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:40.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:40.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:40.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:40.074+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:39.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:39.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:39.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:39.932+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:41.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.055+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:48:41.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.055+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:48:41.057 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:40.685+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:41.057 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:40.685+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:41.057 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:40.685+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:41.057 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:40.686+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:41.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:40.885+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:41.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:40.885+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:41.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:40.885+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:41.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:40.886+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:41.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:41.102+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:41.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:41.102+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:41.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:41.102+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:41.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:41.103+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:41.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.740+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:48:41.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.740+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:48:41.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.740+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f17d4003470 tx=0x7f17e8000f40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:48:41.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.740+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f17d4003470 tx=0x7f17e8000f40 comp rx=0 tx=0).stop 2024-09-10T03:48:41.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.740+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:48:41.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.740+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:41.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.740+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:41.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.740+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:41.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.740+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d805a5e0 2024-09-10T03:48:41.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.740+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:48:41.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.741+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:48:41.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.741+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:48:41.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.741+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:41.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.741+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048750 con 0x7f17d805a5e0 2024-09-10T03:48:41.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.742+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f17e8057040 tx=0x7f17e004d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:48:41.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.742+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8001a50 con 0x7f17d805a5e0 2024-09-10T03:48:41.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.742+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8035be0 con 0x7f17d805a5e0 2024-09-10T03:48:41.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:41.744+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c040 con 0x7f17d805a5e0 2024-09-10T03:48:42.077 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:48:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:48:41.740+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 1291 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:48:42.077 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:41.646+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:42.077 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:41.646+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:42.077 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:41.646+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:42.077 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:41.646+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:42.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:41.865+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:42.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:41.865+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:42.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:41.865+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:42.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:41.866+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:42.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:42.074+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:42.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:42.074+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:42.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:42.074+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:42.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:42.075+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:42.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:42.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:42.834 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:42.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:42.834 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:42.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:42.834 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:42.668+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:43.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:42.900+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:42.900+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:42.900+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:42.901+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:43.121+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:43.121+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:43.121+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:43.122+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:43.701+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:43.701+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:43.701+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:43.702+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:44.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:44.085+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:44.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:44.085+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:44.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:44.085+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:44.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:44.086+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:43.896+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:43.896+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:43.896+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:43.896+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:44.853 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:44.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:44.853 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:44.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:44.853 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:44.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:44.853 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:44.687+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:45.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:45.091+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:45.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:45.091+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:45.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:45.091+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:45.092+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:44.850+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:44.850+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:44.850+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:44.851+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:46.046 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:45.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:46.047 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:45.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:46.047 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:45.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:46.047 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:45.833+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:46.047 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:45.651+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:46.047 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:45.651+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:46.047 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:45.651+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:46.047 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:45.651+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:46.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:46.045+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:46.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:46.045+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:46.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:46.045+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:46.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:46.045+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:46.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.740+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:48:46.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.740+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:48:46.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.740+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f17e8057040 tx=0x7f17e004d000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:48:46.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.740+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f17e8057040 tx=0x7f17e004d000 comp rx=0 tx=0).stop 2024-09-10T03:48:46.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.741+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:48:46.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.741+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:46.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.741+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:46.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.741+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:46.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.741+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059de0 con 0x7f17d8047ae0 2024-09-10T03:48:46.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.741+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:48:46.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.741+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:48:46.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.741+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:48:46.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.742+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:46.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.742+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048aa0 con 0x7f17d8047ae0 2024-09-10T03:48:46.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.742+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f17e801af80 tx=0x7f17e80457f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:48:46.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.742+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e803c310 con 0x7f17d8047ae0 2024-09-10T03:48:46.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.742+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8035be0 con 0x7f17d8047ae0 2024-09-10T03:48:46.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:46.744+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c040 con 0x7f17d8047ae0 2024-09-10T03:48:46.859 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:48:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:48:46.741+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 1353 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:48:46.859 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:46.604+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:46.859 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:46.604+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:46.859 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:46.604+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:46.859 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:46.604+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:47.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:46.856+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:47.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:46.856+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:47.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:46.856+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:47.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:46.857+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:47.334 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:47.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:47.334 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:47.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:47.334 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:47.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:47.334 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:47.088+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:47.560+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:47.560+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:47.560+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:47.561+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:48.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:48.046+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:48.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:48.046+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:48.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:48.046+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:48.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:48.047+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:47.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:47.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:47.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:47.909+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:48.941 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:48.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:48.941 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:48.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:48.941 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:48.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:48.941 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:48.611+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:49.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:48.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:49.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:48.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:49.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:48.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:48.940+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:49.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:49.095+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:49.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:49.095+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:49.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:49.095+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:49.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:49.095+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:49.910 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:49.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:49.910 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:49.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:49.910 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:49.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:49.910 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:49.909+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:49.911 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:49.658+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:49.911 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:49.658+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:49.911 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:49.658+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:49.911 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:49.659+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:50.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:50.049+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:50.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:50.049+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:50.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:50.049+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:50.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:50.049+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:50.958 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:50.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:50.959 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:50.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:50.959 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:50.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:50.959 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:50.684+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:50.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:50.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:50.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:50.958+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:51.078+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:51.078+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:51.078+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:51.079+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:51.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.742+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:48:51.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.742+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:48:51.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.743+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f17e801af80 tx=0x7f17e80457f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:48:51.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.743+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f17e801af80 tx=0x7f17e80457f0 comp rx=0 tx=0).stop 2024-09-10T03:48:51.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.743+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:48:51.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.743+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:51.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.743+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:51.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.743+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:51.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.743+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059de0 con 0x7f17d805a5e0 2024-09-10T03:48:51.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.743+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:48:51.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.743+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:48:51.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.744+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:48:51.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.744+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:51.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.744+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048750 con 0x7f17d805a5e0 2024-09-10T03:48:51.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.744+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f17d8008fe0 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:48:51.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.744+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8026680 con 0x7f17d805a5e0 2024-09-10T03:48:51.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.745+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d805a5e0 2024-09-10T03:48:51.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:51.745+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17d805a5e0 2024-09-10T03:48:51.938 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:48:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:48:51.743+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 1416 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:48:51.938 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:51.640+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:51.938 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:51.640+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:51.938 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:51.640+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:51.938 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:51.641+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:52.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:51.936+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:51.936+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:51.936+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:51.937+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:52.063+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:52.063+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:52.063+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:52.064+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:52.947 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:52.678+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:52.948 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:52.678+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:52.948 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:52.678+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:52.948 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:52.679+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:52.946+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:52.946+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:52.946+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:52.947+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:53.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:53.082+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:53.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:53.082+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:53.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:53.082+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:53.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:53.083+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:53.680+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:53.680+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:53.681+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:53.681+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:53.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:53.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:53.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:53.905+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:54.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:54.072+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:54.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:54.072+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:54.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:54.072+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:54.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:54.072+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:54.912 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:54.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:54.912 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:54.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:54.912 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:54.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:54.912 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:54.730+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:54.910+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:54.910+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:54.910+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:54.911+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:55.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:55.071+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:55.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:55.071+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:55.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:55.071+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:55.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:55.071+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:56.056 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:55.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:56.056 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:55.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:56.056 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:55.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:56.056 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:55.699+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:56.056 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:55.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:56.056 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:55.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:56.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:55.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:56.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:55.895+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:56.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:56.054+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:56.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:56.054+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:56.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:56.054+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:56.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:56.055+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:56.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.543+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:48:56.544 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.543+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:48:56.744 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.743+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:48:56.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.743+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:48:56.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.743+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f17d8008fe0 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:48:56.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.743+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f17d8008fe0 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:48:56.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.743+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:48:56.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.743+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:56.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.743+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:56.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.743+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:48:56.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.743+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059de0 con 0x7f17d8047ae0 2024-09-10T03:48:56.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.743+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:48:56.745 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.744+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:48:56.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.744+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:48:56.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.744+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:48:56.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.744+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048aa0 con 0x7f17d8047ae0 2024-09-10T03:48:56.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.745+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f17e00042d0 tx=0x7f17e8039fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:48:56.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.745+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c050 con 0x7f17d8047ae0 2024-09-10T03:48:56.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.745+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e001ecc0 con 0x7f17d8047ae0 2024-09-10T03:48:56.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:48:56.746+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8047ae0 2024-09-10T03:48:56.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:48:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:48:56.743+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 1479 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:48:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:56.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:56.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:56.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:56.743+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:56.847+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:56.847+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:56.847+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:56.848+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:57.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:57.005+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:57.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:57.005+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:57.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:57.005+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:57.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:57.006+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:58.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:57.767+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:58.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:57.767+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:57.767+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:57.768+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:57.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:57.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:57.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:57.882+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:58.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:58.002+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:58.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:58.002+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:58.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:58.002+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:58.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:58.003+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:59.014 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:58.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:59.015 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:58.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:59.015 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:58.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:59.015 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:58.733+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:59.015 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:58.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:59.015 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:58.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:59.015 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:58.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:48:59.015 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:58.913+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:48:59.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:59.013+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:59.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:59.013+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:59.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:59.013+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:48:59.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:48:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:48:59.014+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:48:59.916 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:59.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:59.916 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:59.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:59.916 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:59.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:48:59.916 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:48:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:48:59.745+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:59.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:59.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:59.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:48:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:48:59.915+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:00.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:00.039+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:00.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:00.039+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:00.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:00.039+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:00.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:00.039+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:01.053 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:00.741+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:01.053 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:00.741+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:01.053 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:00.741+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:01.053 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:00.742+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:01.054 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:00.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:01.054 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:00.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:01.054 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:00.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:01.054 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:00.898+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:01.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:01.052+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:01.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:01.052+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:01.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:01.052+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:01.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:01.052+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:01.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.744+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:49:01.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.744+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:49:01.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.744+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f17e00042d0 tx=0x7f17e8039fb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:49:01.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.744+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f17e00042d0 tx=0x7f17e8039fb0 comp rx=0 tx=0).stop 2024-09-10T03:49:01.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.744+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:49:01.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.744+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:01.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.744+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:01.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.744+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:01.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.744+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059de0 con 0x7f17d805a5e0 2024-09-10T03:49:01.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.744+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:49:01.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.744+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:49:01.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.744+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:49:01.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.744+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:01.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.744+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d4005b00 con 0x7f17d805a5e0 2024-09-10T03:49:01.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.745+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009520 tx=0x7f17e0011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:49:01.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.745+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8026e20 con 0x7f17d805a5e0 2024-09-10T03:49:01.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.745+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8033e70 con 0x7f17d805a5e0 2024-09-10T03:49:01.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:01.747+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17d805a5e0 2024-09-10T03:49:02.042 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:01.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:02.042 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:01.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:02.042 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:01.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:02.042 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:01.758+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:02.043 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:01.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:02.043 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:01.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:02.043 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:01.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:02.043 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:01.925+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:02.043 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:49:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:49:01.744+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 1542 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:49:02.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:02.041+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:02.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:02.041+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:02.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:02.041+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:02.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:02.041+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:03.073 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:02.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:03.073 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:02.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:03.073 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:02.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:03.073 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:02.771+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:03.073 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:02.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:03.073 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:02.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:03.073 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:02.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:03.073 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:02.896+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:03.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:03.071+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:03.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:03.071+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:03.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:03.071+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:03.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:03.071+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:03.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:03.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:03.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:03.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:03.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:03.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:03.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:03.773+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:04.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:04.037+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:04.037+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:04.037+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:04.038+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:03.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:03.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:03.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:03.882+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:04.915 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:04.745+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:04.915 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:04.745+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:04.916 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:04.745+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:04.916 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:04.746+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:05.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:05.044+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:05.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:05.044+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:05.044+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:05.045+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:04.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:04.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:04.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:04.915+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:06.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:05.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:06.034 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:05.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:06.034 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:05.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:06.034 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:05.766+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:06.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:05.937+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:06.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:05.937+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:06.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:05.937+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:06.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:05.938+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:06.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:06.032+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:06.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:06.032+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:06.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:06.032+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:06.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:06.033+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:06.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.744+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:49:06.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.744+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:49:06.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.744+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009520 tx=0x7f17e0011e80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:49:06.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.744+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009520 tx=0x7f17e0011e80 comp rx=0 tx=0).stop 2024-09-10T03:49:06.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.744+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:49:06.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.744+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:06.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.744+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:06.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.744+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:06.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.744+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059de0 con 0x7f17d8047ae0 2024-09-10T03:49:06.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.744+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:49:06.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.745+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:49:06.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.745+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:49:06.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.745+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:06.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.745+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048aa0 con 0x7f17d8047ae0 2024-09-10T03:49:06.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.746+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f17e8021000 tx=0x7f17e80093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:49:06.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.746+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e80331c0 con 0x7f17d8047ae0 2024-09-10T03:49:06.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.746+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d8047ae0 2024-09-10T03:49:06.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:06.747+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d8047ae0 2024-09-10T03:49:07.074 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:06.811+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:07.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:06.811+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:07.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:06.811+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:07.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:06.812+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:07.075 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:49:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:49:06.745+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 1605 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:49:07.075 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:06.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:07.075 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:06.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:07.075 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:06.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:07.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:06.905+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:07.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:07.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:07.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:07.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:07.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:07.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:07.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:07.074+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:08.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:07.859+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:08.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:07.859+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:08.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:07.859+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:07.860+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:08.045+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:08.045+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:08.045+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:08.046+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:08.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:07.877+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:08.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:07.877+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:08.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:07.877+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:08.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:07.878+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:09.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:09.056+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:09.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:09.056+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:09.056+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:09.057+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:08.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:08.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:08.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:08.830+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:09.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:08.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:09.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:08.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:09.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:08.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:09.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:08.905+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:09.808+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:09.808+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:09.808+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:09.808+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:10.027+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:10.027+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:10.027+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:10.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:10.028+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:10.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:09.880+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:10.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:09.880+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:10.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:09.880+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:10.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:09.881+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:11.050 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:10.760+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:11.051 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:10.760+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:11.051 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:10.760+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:11.051 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:10.760+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:11.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:10.840+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:11.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:10.840+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:11.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:10.840+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:11.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:10.841+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:11.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:11.049+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:11.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:11.049+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:11.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:11.049+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:11.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:11.050+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:11.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.745+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:49:11.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.745+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:49:11.746 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.745+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f17e8021000 tx=0x7f17e80093a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:49:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.745+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f17e8021000 tx=0x7f17e80093a0 comp rx=0 tx=0).stop 2024-09-10T03:49:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.745+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:49:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.745+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.745+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.745+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.745+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059de0 con 0x7f17d805a5e0 2024-09-10T03:49:11.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.745+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:49:11.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.745+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:49:11.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.746+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:49:11.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.746+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:11.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.746+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048750 con 0x7f17d805a5e0 2024-09-10T03:49:11.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.746+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f17e000dfd0 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:49:11.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.747+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028030 con 0x7f17d805a5e0 2024-09-10T03:49:11.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.747+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0057e80 con 0x7f17d805a5e0 2024-09-10T03:49:11.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.748+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d805a5e0 2024-09-10T03:49:11.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.903+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:49:11.904 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:11.903+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:49:12.048 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:49:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:49:11.745+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 1668 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:49:12.048 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:11.721+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:12.049 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:11.721+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:12.049 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:11.721+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:12.049 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:11.722+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:12.049 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:11.876+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:12.049 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:11.876+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:12.049 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:11.876+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:12.049 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:11.877+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:12.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:12.047+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:12.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:12.047+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:12.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:12.047+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:12.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:12.048+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:13.011 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:12.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:13.011 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:12.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:13.011 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:12.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:13.011 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:12.710+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:13.011 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:12.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:13.011 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:12.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:13.011 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:12.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:13.011 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:12.895+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:13.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:13.009+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:13.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:13.009+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:13.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:13.009+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:13.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:13.010+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:13.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:13.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:13.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:13.696+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:13.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:13.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:13.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:13.882+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:14.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:13.977+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:14.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:13.977+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:14.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:13.977+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:14.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:13.977+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:14.894 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:14.688+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:14.895 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:14.688+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:14.895 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:14.688+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:14.895 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:14.689+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:14.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:14.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:14.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:14.894+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:14.991+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:14.991+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:14.991+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:14.992+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:16.031 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:15.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:16.032 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:15.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:16.032 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:15.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:16.032 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:15.666+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:16.032 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:15.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:16.032 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:15.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:16.032 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:15.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:16.032 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:15.893+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:16.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:16.030+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:16.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:16.030+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:16.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:16.030+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:16.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:16.031+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:16.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.746+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:49:16.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.746+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:49:16.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.746+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f17e000dfd0 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:49:16.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.746+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f17e000dfd0 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:49:16.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.746+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:49:16.747 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.746+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:16.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.746+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:16.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.746+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:16.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.746+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059de0 con 0x7f17d8047ae0 2024-09-10T03:49:16.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.746+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:49:16.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.746+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:49:16.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.747+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:49:16.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.747+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:16.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.747+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048750 con 0x7f17d8047ae0 2024-09-10T03:49:16.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.747+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f17d800f8c0 tx=0x7f17e0050fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:49:16.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.747+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c040 con 0x7f17d8047ae0 2024-09-10T03:49:16.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.748+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e005de10 con 0x7f17d8047ae0 2024-09-10T03:49:16.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:16.749+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8047ae0 2024-09-10T03:49:17.033 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:49:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:49:16.746+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 1731 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:49:17.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:16.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:17.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:16.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:17.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:16.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:17.034 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:16.656+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:17.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:16.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:17.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:16.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:17.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:16.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:17.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:16.891+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:17.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:17.031+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:17.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:17.031+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:17.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:17.031+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:17.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:17.032+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:18.048 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:17.876+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:18.048 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:17.876+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:18.048 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:17.876+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:18.048 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:17.876+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:18.048 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:17.652+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:18.048 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:17.652+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:18.048 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:17.652+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:18.049 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:17.653+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:18.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:18.046+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:18.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:18.046+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:18.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:18.046+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:18.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:18.047+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:18.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:18.873+0000 7f17de7fc640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f17d4005dc0 con 0x7f17d8047ae0 2024-09-10T03:49:19.071 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:18.829+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:19.071 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:18.829+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:19.071 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:18.829+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:19.071 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:18.830+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:19.071 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:18.663+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:19.071 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:18.663+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:19.071 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:18.663+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:19.072 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:18.663+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:19.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:19.068+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:19.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:19.068+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:19.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:19.068+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:19.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:19.069+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:19.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:19.873+0000 7f17de7fc640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f17d4005ad0 con 0x7f17d8047ae0 2024-09-10T03:49:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:19.688+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:19.688+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:19.688+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:19.688+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:19.834+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:19.834+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:19.834+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:19.835+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:20.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:20.110+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:20.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:20.110+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:20.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:20.110+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:20.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:20.111+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:20.875 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:20.873+0000 7f17de7fc640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f17d4005ad0 con 0x7f17d8047ae0 2024-09-10T03:49:21.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:20.703+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:21.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:20.703+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:21.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:20.703+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:20.703+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:20.848+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:20.848+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:20.848+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:20.849+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:21.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:21.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:21.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:21.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:21.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:21.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:21.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:21.133+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:21.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.745+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:49:21.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.745+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:49:21.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.745+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f17d800f8c0 tx=0x7f17e0050fc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:49:21.748 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.745+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f17d800f8c0 tx=0x7f17e0050fc0 comp rx=0 tx=0).stop 2024-09-10T03:49:21.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.747+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:49:21.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.747+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:21.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.747+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:21.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.747+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:21.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.747+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059dc0 con 0x7f17d805a5e0 2024-09-10T03:49:21.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.747+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:49:21.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.747+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:49:21.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.747+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:49:21.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.747+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:21.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.747+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048aa0 con 0x7f17d805a5e0 2024-09-10T03:49:21.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.748+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f17e003f250 tx=0x7f17e0009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:49:21.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.748+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c020 con 0x7f17d805a5e0 2024-09-10T03:49:21.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.748+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0004340 con 0x7f17d805a5e0 2024-09-10T03:49:21.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:21.749+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e003c210 con 0x7f17d805a5e0 2024-09-10T03:49:22.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:21.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:21.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:21.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:21.895+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:21.659+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:21.659+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:21.659+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:21.660+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:22.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:49:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:49:21.746+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 1794 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:49:22.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:22.165+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:22.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:22.165+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:22.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:22.165+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:22.166+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:22.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:22.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:22.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:22.685+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:22.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:22.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:22.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:22.879+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:23.539 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:23.165+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:23.539 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:23.165+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:23.540 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:23.165+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:23.540 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:23.165+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:23.668+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:23.668+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:23.668+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:23.669+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:24.189 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:23.840+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:24.190 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:23.840+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:24.190 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:23.840+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:24.190 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:23.840+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:24.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:24.187+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:24.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:24.187+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:24.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:24.187+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:24.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:24.188+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:24.890 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:24.659+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:24.891 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:24.659+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:24.891 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:24.659+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:24.891 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:24.660+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:24.891 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:24.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:24.891 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:24.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:24.891 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:24.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:25.223 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:24.890+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:25.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:25.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:25.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:25.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:25.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:25.222+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:25.930 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:25.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:25.930 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:25.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:25.930 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:25.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:25.930 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:25.648+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:26.184 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:25.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:26.184 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:25.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:26.184 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:25.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:26.184 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:25.929+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:26.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:26.182+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:26.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:26.182+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:26.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:26.182+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:26.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:26.183+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:26.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.747+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:49:26.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.747+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:49:26.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.747+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f17e003f250 tx=0x7f17e0009190 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:49:26.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.747+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f17e003f250 tx=0x7f17e0009190 comp rx=0 tx=0).stop 2024-09-10T03:49:26.749 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.747+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:49:26.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.747+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:26.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.747+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:26.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.747+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:26.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.747+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d8047ae0 2024-09-10T03:49:26.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.747+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:49:26.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.748+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:49:26.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.748+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:49:26.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.748+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:26.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.748+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048920 con 0x7f17d8047ae0 2024-09-10T03:49:26.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.749+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f17e8031f50 tx=0x7f17e0025000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:49:26.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.750+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e801d5b0 con 0x7f17d8047ae0 2024-09-10T03:49:26.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.750+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803bec0 con 0x7f17d8047ae0 2024-09-10T03:49:26.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:26.751+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d8047ae0 2024-09-10T03:49:26.934 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:49:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:49:26.747+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 1856 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:49:26.934 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:26.671+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:26.934 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:26.671+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:26.934 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:26.671+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:26.934 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:26.672+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:27.198 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:26.932+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:27.198 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:26.932+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:27.198 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:26.932+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:27.199 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:26.933+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:27.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:27.263+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:49:27.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:27.263+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:49:27.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:27.197+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:27.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:27.197+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:27.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:27.197+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:27.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:27.198+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:28.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:27.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:28.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:27.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:27.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:27.684+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:27.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:27.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:27.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:27.919+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:28.423 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:28.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:28.423 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:28.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:28.423 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:28.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:28.423 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:28.152+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:29.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:28.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:28.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:28.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:28.712+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:28.900+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:28.900+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:28.900+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:28.901+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:29.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:29.196+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:29.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:29.196+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:29.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:29.196+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:29.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:29.197+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:29.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:29.875+0000 7f17de7fc640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f17d4005a90 con 0x7f17d8047ae0 2024-09-10T03:49:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:29.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:29.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:29.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:29.743+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:29.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:29.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:29.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:29.932+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:30.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:30.150+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:30.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:30.150+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:30.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:30.150+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:30.151+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:30.876 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:30.875+0000 7f17de7fc640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f17d4005e60 con 0x7f17d8047ae0 2024-09-10T03:49:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:30.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:30.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:30.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:30.775+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:30.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:30.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:30.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:30.979+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:31.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:31.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:31.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:31.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:31.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:31.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:31.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:31.152+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:31.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.748+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:49:31.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.748+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:49:31.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.748+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f17e8031f50 tx=0x7f17e0025000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:49:31.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.748+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f17e8031f50 tx=0x7f17e0025000 comp rx=0 tx=0).stop 2024-09-10T03:49:31.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.748+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:49:31.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.748+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:31.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.748+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:31.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.748+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:31.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.748+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804f590 con 0x7f17d805a5e0 2024-09-10T03:49:31.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.748+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:49:31.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.748+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:49:31.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.749+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:49:31.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.749+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:31.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.749+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048920 con 0x7f17d805a5e0 2024-09-10T03:49:31.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.750+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f17e002dc70 tx=0x7f17e004d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:49:31.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.751+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028070 con 0x7f17d805a5e0 2024-09-10T03:49:31.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.751+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e001bca0 con 0x7f17d805a5e0 2024-09-10T03:49:31.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:31.752+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d805a5e0 2024-09-10T03:49:32.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:31.776+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:32.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:31.776+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:31.776+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:31.777+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:32.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:49:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:49:31.748+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 1919 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:49:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:31.953+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:31.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:31.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:31.954+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:32.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:32.168+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:32.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:32.168+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:32.168+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:32.169+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:32.784+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:32.784+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:32.784+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:32.785+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:33.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:32.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:33.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:32.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:33.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:32.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:33.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:32.913+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:33.533 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:33.193+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:33.534 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:33.193+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:33.534 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:33.193+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:33.534 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:33.194+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:34.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:33.811+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:34.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:33.811+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:34.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:33.811+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:34.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:33.812+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:33.958+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:33.958+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:33.958+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:33.959+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:34.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:34.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:34.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:34.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:34.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:34.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:34.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:34.240+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:35.197 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:34.854+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:35.197 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:34.854+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:35.197 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:34.854+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:35.197 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:34.855+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:35.197 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:34.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:35.198 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:34.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:35.198 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:34.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:35.198 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:34.958+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:35.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:35.195+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:35.195+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:35.195+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:35.196+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:36.171 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:35.853+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:36.172 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:35.853+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:36.172 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:35.853+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:36.172 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:35.853+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:36.172 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:35.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:36.172 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:35.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:36.172 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:35.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:36.172 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:35.949+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:36.480 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:36.170+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:36.480 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:36.170+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:36.480 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:36.170+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:36.480 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:36.171+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:36.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.749+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:49:36.750 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.749+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:49:36.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.749+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f17e002dc70 tx=0x7f17e004d000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:49:36.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.749+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f17e002dc70 tx=0x7f17e004d000 comp rx=0 tx=0).stop 2024-09-10T03:49:36.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.749+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:49:36.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.749+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:36.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.749+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:36.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.749+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:36.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.749+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804f590 con 0x7f17d8047ae0 2024-09-10T03:49:36.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.749+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:49:36.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.749+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:49:36.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.750+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:49:36.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.750+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:36.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.750+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048750 con 0x7f17d8047ae0 2024-09-10T03:49:36.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.750+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f17e80421f0 tx=0x7f17e00532a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:49:36.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.750+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055030 con 0x7f17d8047ae0 2024-09-10T03:49:36.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.751+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b030 con 0x7f17d8047ae0 2024-09-10T03:49:36.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:36.752+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c100 con 0x7f17d8047ae0 2024-09-10T03:49:37.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:49:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:49:36.749+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 1982 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:49:37.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:36.827+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:36.827+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:36.827+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:36.828+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:36.986+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:36.986+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:36.986+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:36.987+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:37.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:37.190+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:37.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:37.190+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:37.190+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:37.191+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:38.180 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:37.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:38.180 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:37.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:38.180 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:37.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:38.180 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:37.837+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:38.181 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:37.990+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:38.181 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:37.990+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:38.181 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:37.990+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:38.181 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:37.991+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:38.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:38.178+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:38.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:38.178+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:38.178+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:38.179+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:38.957 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:38.830+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:38.957 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:38.830+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:38.957 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:38.830+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:38.957 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:38.831+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:39.228 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:38.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:39.228 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:38.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:39.228 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:38.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:39.228 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:38.956+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:39.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:39.226+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:39.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:39.226+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:39.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:39.226+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:39.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:39.227+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:40.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:39.787+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:39.787+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:39.787+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:39.787+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:40.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:39.959+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:40.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:39.959+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:40.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:39.959+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:40.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:39.960+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:40.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:40.257+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:40.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:40.257+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:40.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:40.257+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:40.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:40.258+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:41.228 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:40.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:41.228 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:40.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:41.228 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:40.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:41.228 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:40.835+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:41.229 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:40.967+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:41.229 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:40.967+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:41.229 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:40.967+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:41.229 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:40.968+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:41.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:41.226+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:41.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:41.226+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:41.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:41.226+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:41.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:41.227+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:41.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.749+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:49:41.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.749+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:49:41.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.749+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f17e80421f0 tx=0x7f17e00532a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:49:41.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.749+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f17e80421f0 tx=0x7f17e00532a0 comp rx=0 tx=0).stop 2024-09-10T03:49:41.751 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.749+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:49:41.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.749+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:41.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.749+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:41.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.749+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:41.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.749+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804f5b0 con 0x7f17d805a5e0 2024-09-10T03:49:41.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.749+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:49:41.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.749+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:49:41.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.750+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:49:41.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.750+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:41.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.750+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048750 con 0x7f17d805a5e0 2024-09-10T03:49:41.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.751+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f17d8012390 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:49:41.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.751+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028050 con 0x7f17d805a5e0 2024-09-10T03:49:41.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.751+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0032ec0 con 0x7f17d805a5e0 2024-09-10T03:49:41.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:41.753+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e003c8c0 con 0x7f17d805a5e0 2024-09-10T03:49:42.008 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:49:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:49:41.749+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 2045 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:49:42.009 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:41.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:42.009 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:41.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:42.009 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:41.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:42.009 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:41.841+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:42.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:42.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:42.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:42.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:42.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:42.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:42.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:42.239+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:42.007+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:42.007+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:42.007+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:42.007+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:42.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:42.623+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:49:42.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:42.623+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:49:42.979 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:42.832+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:42.980 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:42.832+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:42.980 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:42.832+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:42.980 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:42.833+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:43.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:43.196+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:43.196+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:43.196+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:43.197+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:42.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:42.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:42.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:42.979+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:44.190 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:43.970+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:44.190 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:43.970+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:44.190 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:43.970+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:44.190 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:43.970+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:44.191 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:43.864+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:44.191 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:43.864+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:44.191 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:43.864+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:44.191 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:43.865+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:44.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:44.188+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:44.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:44.188+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:44.188+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:44.189+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:44.993 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:44.843+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:44.993 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:44.843+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:44.993 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:44.844+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:44.993 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:44.844+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:45.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:45.165+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:45.165+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:45.165+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:45.166+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:44.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:44.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:44.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:44.992+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:46.037 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:45.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:46.037 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:45.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:46.037 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:45.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:46.037 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:45.875+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:46.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:46.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:46.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:46.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:46.035+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:46.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:46.210+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:46.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:46.210+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:46.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:46.210+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:46.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:46.210+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:46.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.749+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:49:46.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.749+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:49:46.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.749+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f17d8012390 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:49:46.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.750+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f17d8012390 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:49:46.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.750+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:49:46.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.750+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:46.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.750+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:46.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.750+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:46.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.750+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804f590 con 0x7f17d8047ae0 2024-09-10T03:49:46.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.750+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:49:46.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.750+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:49:46.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.750+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:49:46.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.750+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:46.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.751+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048750 con 0x7f17d8047ae0 2024-09-10T03:49:46.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.751+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f17e8014010 tx=0x7f17e0050fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:49:46.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.752+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055050 con 0x7f17d8047ae0 2024-09-10T03:49:46.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.752+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d8047ae0 2024-09-10T03:49:46.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:46.753+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17d8047ae0 2024-09-10T03:49:47.004 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:49:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:49:46.750+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 2108 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:49:47.004 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:46.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:47.004 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:46.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:47.004 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:46.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:47.004 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:46.909+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:47.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:47.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:47.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:47.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:47.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:47.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:47.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:47.003+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:47.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:47.258+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:47.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:47.258+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:47.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:47.258+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:47.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:47.259+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:48.295 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:47.937+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:48.296 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:47.937+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:48.296 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:47.937+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:48.296 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:47.937+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:48.296 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:47.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:48.296 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:47.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:48.296 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:47.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:48.296 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:47.955+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:48.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:48.294+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:48.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:48.294+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:48.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:48.294+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:48.295+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:49.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:48.912+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:49.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:48.912+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:48.912+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:48.912+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:48.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:48.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:48.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:48.958+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:49.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:49.269+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:49.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:49.269+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:49.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:49.269+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:49.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:49.270+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:50.227 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:49.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:50.227 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:49.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:50.227 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:49.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:50.227 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:49.964+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:50.228 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:49.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:50.228 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:49.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:50.228 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:49.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:50.228 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:49.934+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:50.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:50.226+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:50.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:50.226+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:50.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:50.226+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:50.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:50.227+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:51.215 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:50.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:51.216 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:50.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:51.216 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:50.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:51.216 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:50.918+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:51.216 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:50.950+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:51.216 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:50.951+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:51.216 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:50.951+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:51.216 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:50.951+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:51.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:51.214+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:51.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:51.214+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:51.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:51.214+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:51.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:51.215+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:51.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.750+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:49:51.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.750+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:49:51.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.750+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f17e8014010 tx=0x7f17e0050fc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:49:51.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.750+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f17e8014010 tx=0x7f17e0050fc0 comp rx=0 tx=0).stop 2024-09-10T03:49:51.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.750+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:49:51.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.750+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:51.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.750+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:51.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.750+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:51.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.750+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804f4e0 con 0x7f17d805a5e0 2024-09-10T03:49:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.750+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:49:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.751+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:49:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.751+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:49:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.751+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.751+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048750 con 0x7f17d805a5e0 2024-09-10T03:49:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.751+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f17e0011040 tx=0x7f17e800bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:49:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.752+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e00159e0 con 0x7f17d805a5e0 2024-09-10T03:49:51.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.752+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e00117f0 con 0x7f17d805a5e0 2024-09-10T03:49:51.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:51.753+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d805a5e0 2024-09-10T03:49:52.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:51.956+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:51.956+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:51.956+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:51.956+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:51.956+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:51.956+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:51.956+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:51.957+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:52.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:49:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:49:51.751+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 2171 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:49:52.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:52.185+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:52.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:52.185+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:52.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:52.185+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:52.185+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:53.215 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:52.959+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:53.215 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:52.959+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:53.215 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:52.959+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:53.215 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:52.959+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:53.215 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:52.969+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:53.215 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:52.969+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:53.215 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:52.969+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:53.215 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:52.969+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:53.541 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:53.213+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:53.541 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:53.213+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:53.541 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:53.213+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:53.541 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:53.214+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:54.246 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:53.991+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:54.246 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:53.991+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:54.246 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:53.991+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:54.246 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:53.992+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:54.247 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:53.946+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:54.247 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:53.946+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:54.247 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:53.946+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:54.247 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:53.947+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:54.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:54.245+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:54.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:54.245+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:54.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:54.245+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:54.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:54.245+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:55.262 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:55.010+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:55.262 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:55.010+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:55.262 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:55.010+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:55.262 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:55.011+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:55.262 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:54.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:55.262 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:54.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:55.262 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:54.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:55.263 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:54.903+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:55.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:55.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:55.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:55.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:55.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:55.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:55.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:55.261+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:56.263 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:55.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:56.263 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:55.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:56.263 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:55.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:56.263 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:55.994+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:56.264 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:55.860+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:56.264 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:55.860+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:56.264 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:55.860+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:56.264 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:55.861+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:56.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:56.261+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:56.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:56.261+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:56.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:56.261+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:56.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:56.261+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:56.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.750+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:49:56.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.750+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:49:56.752 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.750+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f17e0011040 tx=0x7f17e800bf00 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:49:56.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.750+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f17e0011040 tx=0x7f17e800bf00 comp rx=0 tx=0).stop 2024-09-10T03:49:56.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.750+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:49:56.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.750+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:56.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.750+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:56.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.750+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:49:56.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.750+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d8047ae0 2024-09-10T03:49:56.753 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.750+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:49:56.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.751+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:49:56.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.751+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:49:56.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.751+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:49:56.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.751+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048920 con 0x7f17d8047ae0 2024-09-10T03:49:56.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.751+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f17e00474c0 tx=0x7f17e005d500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:49:56.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.752+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c8c0 con 0x7f17d8047ae0 2024-09-10T03:49:56.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.752+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e000dec0 con 0x7f17d8047ae0 2024-09-10T03:49:56.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:56.753+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d8047ae0 2024-09-10T03:49:57.011 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:49:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:49:56.751+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 2234 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:49:57.012 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:56.851+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:57.012 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:56.851+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:57.012 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:56.851+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:57.012 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:56.852+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:57.287 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:57.009+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:57.288 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:57.009+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:57.288 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:57.009+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:57.288 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:57.010+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:57.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:57.285+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:57.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:57.285+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:57.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:57.285+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:57.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:57.286+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:57.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:57.982+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:49:57.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:49:57.982+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:49:58.255 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:58.032+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:58.255 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:58.032+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:58.255 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:58.032+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:58.255 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:58.033+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:58.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:57.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:58.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:57.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:58.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:57.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:58.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:57.845+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:58.252+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:58.252+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:58.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:58.252+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:58.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:58.253+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:49:59.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:59.019+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:59.019+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:59.019+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:49:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:49:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:49:59.020+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:58.801+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:58.801+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:58.801+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:49:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:58.802+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:49:59.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:59.250+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:59.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:59.250+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:59.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:59.250+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:49:59.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:49:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:49:59.250+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:00.059 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:59.761+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:00.059 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:59.761+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:00.059 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:59.762+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:00.059 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:49:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:49:59.762+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:00.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:00.300+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:00.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:00.300+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:00.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:00.300+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:00.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:00.301+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:00.057+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:00.057+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:00.057+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:00.057+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:01.010 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:00.719+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:01.010 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:00.719+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:01.010 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:00.719+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:01.010 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:00.719+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:01.316 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:01.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:01.317 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:01.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:01.317 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:01.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:01.317 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:01.008+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:01.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:01.315+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:01.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:01.315+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:01.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:01.315+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:01.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:01.315+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:01.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.752+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:50:01.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.752+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:50:01.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.752+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f17e00474c0 tx=0x7f17e005d500 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:50:01.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.752+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f17e00474c0 tx=0x7f17e005d500 comp rx=0 tx=0).stop 2024-09-10T03:50:01.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.752+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:50:01.754 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.752+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:01.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.752+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:01.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.752+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:01.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.752+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d805a5e0 2024-09-10T03:50:01.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.752+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:50:01.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.753+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:50:01.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.754+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:50:01.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.754+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:01.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.754+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048920 con 0x7f17d805a5e0 2024-09-10T03:50:01.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.754+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f17e8024010 tx=0x7f17e0011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:50:01.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.755+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055050 con 0x7f17d805a5e0 2024-09-10T03:50:01.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.755+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d805a5e0 2024-09-10T03:50:01.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:01.756+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17d805a5e0 2024-09-10T03:50:02.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:02.000+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:02.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:02.000+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:02.000+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:02.001+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:01.756+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:01.756+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:01.756+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:01.757+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:02.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:50:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:50:01.752+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 2297 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:50:02.732 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:02.352+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:02.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:02.352+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:02.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:02.352+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:02.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:02.353+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:02.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:02.731+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:03.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:02.731+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:03.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:02.731+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:03.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:02.732+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:02.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:02.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:02.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:02.959+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:03.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:03.335+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:03.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:03.335+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:03.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:03.335+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:03.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:03.336+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:04.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:03.780+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:04.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:03.780+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:04.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:03.780+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:04.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:03.780+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:03.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:03.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:03.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:03.996+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:04.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:04.325+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:04.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:04.325+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:04.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:04.325+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:04.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:04.326+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:05.029 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:04.815+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:05.029 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:04.815+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:05.029 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:04.815+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:05.029 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:04.816+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:05.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:05.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:05.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:05.028+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:05.771 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:05.352+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:05.771 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:05.352+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:05.771 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:05.352+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:05.772 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:05.352+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:06.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:05.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:06.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:05.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:06.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:05.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:05.770+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:05.983+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:05.983+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:05.983+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:05.984+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:06.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.753+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:50:06.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.753+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:50:06.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.753+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f17e8024010 tx=0x7f17e0011e80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:50:06.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.753+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f17e8024010 tx=0x7f17e0011e80 comp rx=0 tx=0).stop 2024-09-10T03:50:06.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.753+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:50:06.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.753+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5e0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:06.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.753+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:06.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.753+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:06.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.753+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d8047ae0 2024-09-10T03:50:06.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:06.388+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:06.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:06.388+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:06.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:06.388+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:06.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:06.389+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:06.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.753+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:50:06.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.754+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:50:06.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.754+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:50:06.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.754+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:06.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.755+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048920 con 0x7f17d8047ae0 2024-09-10T03:50:06.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.755+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f17d8011f40 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:50:06.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.755+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055040 con 0x7f17d8047ae0 2024-09-10T03:50:06.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.756+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8033c40 con 0x7f17d8047ae0 2024-09-10T03:50:06.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:06.757+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c040 con 0x7f17d8047ae0 2024-09-10T03:50:07.019 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:50:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:50:06.753+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 2361 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:50:07.019 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:06.761+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:07.019 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:06.761+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:07.019 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:06.761+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:07.019 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:06.762+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:07.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:07.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:07.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:07.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:07.018+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:07.772 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:07.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:07.772 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:07.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:07.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:07.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:07.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:07.375+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:08.000+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:08.000+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:08.000+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:08.001+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:07.767+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:07.767+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:07.767+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:07.768+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:08.766 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:08.366+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:08.766 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:08.366+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:08.766 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:08.366+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:08.766 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:08.366+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:09.018 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:08.764+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:09.018 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:08.764+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:09.018 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:08.764+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:09.018 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:08.765+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:09.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:09.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:09.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:09.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:09.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:09.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:09.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:09.017+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:09.796 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:09.381+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:09.796 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:09.381+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:09.797 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:09.381+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:09.797 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:09.381+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:10.047 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:09.795+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:10.048 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:09.795+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:10.048 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:09.795+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:10.048 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:09.795+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:10.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:10.046+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:10.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:10.046+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:10.046+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:10.047+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:10.816 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:10.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:10.816 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:10.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:10.816 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:10.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:10.816 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:10.396+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:11.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:11.033+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:11.033+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:11.033+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:11.034+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:11.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:10.814+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:11.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:10.814+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:11.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:10.814+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:11.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:10.815+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:11.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.754+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:50:11.755 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.754+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:50:11.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.754+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f17d8011f40 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:50:11.756 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.754+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f17d8011f40 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:50:11.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:11.424+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:11.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:11.424+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:11.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:11.424+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:11.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:11.425+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:11.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.754+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:50:11.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.754+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:11.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.754+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:11.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.754+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.754+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d805a5c0 2024-09-10T03:50:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.754+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:50:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.755+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:50:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.755+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:50:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.755+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.755+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048920 con 0x7f17d805a5c0 2024-09-10T03:50:11.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.755+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f17d800fcb0 tx=0x7f17e800b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:50:11.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.756+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055040 con 0x7f17d805a5c0 2024-09-10T03:50:11.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.756+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d805a5c0 2024-09-10T03:50:11.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:11.757+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c040 con 0x7f17d805a5c0 2024-09-10T03:50:12.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:50:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:50:11.754+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 2424 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:50:12.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:11.813+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:11.813+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:11.813+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:11.814+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:12.438 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:12.084+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:12.439 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:12.084+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:12.439 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:12.084+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:12.439 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:12.085+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:12.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:12.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:12.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:12.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:12.438+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:13.092 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:12.833+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:13.092 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:12.833+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:13.092 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:12.833+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:13.092 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:12.834+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:13.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:13.343+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:50:13.344 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:13.343+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:50:13.344 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:13.090+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:13.344 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:13.090+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:13.345 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:13.090+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:13.345 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:13.091+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:13.484+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:13.484+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:13.484+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:13.485+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:14.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:14.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:14.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:14.062+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:14.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:13.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:14.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:13.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:14.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:13.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:14.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:13.871+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:14.491+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:14.491+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:14.491+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:14.491+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:15.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:15.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:15.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:15.043+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:14.915+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:14.915+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:14.915+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:14.915+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:15.457+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:15.457+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:15.457+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:15.458+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:16.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:16.019+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:16.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:16.019+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:16.019+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:16.019+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:16.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:15.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:16.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:15.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:16.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:15.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:16.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:15.931+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:16.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.755+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:50:16.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.755+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:50:16.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.755+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f17d800fcb0 tx=0x7f17e800b700 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:50:16.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.755+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f17d800fcb0 tx=0x7f17e800b700 comp rx=0 tx=0).stop 2024-09-10T03:50:16.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.755+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:50:16.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.755+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:16.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.755+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:16.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.755+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:16.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.755+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d8047ae0 2024-09-10T03:50:16.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.755+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:50:16.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:16.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:16.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:16.498+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:16.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:16.498+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:16.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:16.498+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:16.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.755+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:50:16.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.756+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:50:16.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.756+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:16.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.756+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048920 con 0x7f17d8047ae0 2024-09-10T03:50:16.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.756+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f17e0041010 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:50:16.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.757+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028070 con 0x7f17d8047ae0 2024-09-10T03:50:16.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.757+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e00117f0 con 0x7f17d8047ae0 2024-09-10T03:50:16.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:16.758+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d8047ae0 2024-09-10T03:50:17.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:16.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:17.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:16.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:17.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:16.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:17.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:16.903+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:17.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:50:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:50:16.755+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 2487 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:50:17.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:17.032+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:17.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:17.032+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:17.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:17.032+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:17.033+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:17.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:17.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:17.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:17.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:17.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:17.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:17.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:17.526+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:18.053+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:18.053+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:18.053+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:18.054+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:18.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:17.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:18.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:17.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:18.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:17.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:18.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:17.951+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:18.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:18.558+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:18.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:18.558+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:18.558+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:18.559+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:19.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:19.042+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:19.042+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:19.042+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:19.043+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:19.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:18.969+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:19.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:18.969+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:19.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:18.969+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:19.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:18.969+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:19.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:19.518+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:19.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:19.518+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:19.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:19.518+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:19.519+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:20.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:20.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:20.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:20.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:20.063+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:19.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:19.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:19.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:20.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:19.926+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:20.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:20.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:20.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:20.517+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:21.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:21.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:21.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:21.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:21.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:21.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:21.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:21.081+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:20.913+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:20.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:20.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:20.914+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:21.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.756+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:50:21.757 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.756+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:50:21.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.756+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f17e0041010 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:50:21.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.756+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f17e0041010 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:50:21.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.756+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:50:21.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.756+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:21.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.756+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:21.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.756+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:21.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.756+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d805a5c0 2024-09-10T03:50:21.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.756+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:50:21.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.757+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:50:21.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.757+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:50:21.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.757+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:21.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.757+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048750 con 0x7f17d805a5c0 2024-09-10T03:50:21.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.757+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009340 tx=0x7f17e0009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:50:21.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.758+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8008b00 con 0x7f17d805a5c0 2024-09-10T03:50:21.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.759+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8033e30 con 0x7f17d805a5c0 2024-09-10T03:50:21.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:21.759+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c040 con 0x7f17d805a5c0 2024-09-10T03:50:21.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:50:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:50:21.756+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 2550 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:50:21.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:21.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:21.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:21.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:21.517+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:22.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:22.041+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:22.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:22.041+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:22.041+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:22.041+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:22.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:21.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:22.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:21.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:22.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:21.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:22.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:21.894+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:22.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:22.508+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:22.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:22.508+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:22.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:22.508+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:22.508+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:23.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:23.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:23.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:23.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:23.059+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:22.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:23.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:22.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:23.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:22.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:23.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:22.908+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:23.521+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:23.521+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:23.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:23.522+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:24.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:24.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:24.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:24.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:24.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:24.029+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:23.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:23.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:24.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:23.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:24.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:23.955+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:24.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:24.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:24.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:24.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:24.523+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:25.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:25.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:25.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:25.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:25.074+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:25.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:24.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:25.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:24.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:25.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:24.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:25.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:24.939+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:25.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:25.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:25.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:25.505+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:26.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:26.119+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:26.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:26.119+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:26.119+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:26.120+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:25.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:25.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:25.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:25.948+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:26.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.756+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:50:26.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.756+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:50:26.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.756+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009340 tx=0x7f17e0009190 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:50:26.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.756+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009340 tx=0x7f17e0009190 comp rx=0 tx=0).stop 2024-09-10T03:50:26.758 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.756+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:50:26.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.756+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:26.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.756+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:26.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.756+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:26.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.756+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804f5b0 con 0x7f17d8047ae0 2024-09-10T03:50:26.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.756+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:50:26.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.757+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:50:26.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.758+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:50:26.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.758+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:26.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.758+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d4005a90 con 0x7f17d8047ae0 2024-09-10T03:50:26.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.758+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f17e802bb30 tx=0x7f17e8017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:50:26.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.760+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8008580 con 0x7f17d8047ae0 2024-09-10T03:50:26.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.760+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8062070 con 0x7f17d8047ae0 2024-09-10T03:50:26.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:26.761+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064070 con 0x7f17d8047ae0 2024-09-10T03:50:26.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:50:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:50:26.757+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 2614 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:50:26.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:26.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:26.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:26.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:26.533+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:27.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:27.141+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:27.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:27.141+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:27.141+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:27.142+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:26.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:26.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:26.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:26.907+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:27.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:27.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:27.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:27.556+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:28.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:28.143+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:28.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:28.143+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:28.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:28.143+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:28.144+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:27.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:27.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:27.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:27.926+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:28.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:28.703+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:50:28.704 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:28.703+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:50:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:28.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:28.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:28.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:28.554+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:29.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:29.114+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:29.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:29.114+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:29.114+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:29.115+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:29.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:28.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:29.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:28.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:29.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:28.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:29.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:28.882+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:29.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:29.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:29.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:29.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:29.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:29.555+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:30.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:30.102+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:30.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:30.102+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:30.102+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:30.103+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:29.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:29.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:29.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:29.858+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:30.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:30.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:30.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:30.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:30.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:30.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:30.895 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:30.586+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:31.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:31.092+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:31.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:31.092+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:31.092+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:31.093+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:30.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:30.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:30.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:30.894+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:31.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.757+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:50:31.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.757+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:50:31.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.757+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f17e802bb30 tx=0x7f17e8017040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:50:31.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.757+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f17e802bb30 tx=0x7f17e8017040 comp rx=0 tx=0).stop 2024-09-10T03:50:31.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.758+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:50:31.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.758+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:31.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.758+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:31.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.758+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:31.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.758+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804f5b0 con 0x7f17d805a5c0 2024-09-10T03:50:31.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.758+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:50:31.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.758+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:50:31.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.759+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:50:31.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.759+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:31.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.759+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d4005a90 con 0x7f17d805a5c0 2024-09-10T03:50:31.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.759+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f17e8008390 tx=0x7f17e8042170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:50:31.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.759+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8001260 con 0x7f17d805a5c0 2024-09-10T03:50:31.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.760+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e802e6c0 con 0x7f17d805a5c0 2024-09-10T03:50:31.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:31.761+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064070 con 0x7f17d805a5c0 2024-09-10T03:50:31.876 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:50:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:50:31.757+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 2678 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:50:31.876 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:31.623+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:31.876 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:31.623+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:31.876 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:31.623+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:31.876 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:31.624+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:32.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:32.100+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:32.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:32.100+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:32.100+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:32.100+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:31.874+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:31.874+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:31.874+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:31.875+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:33.074 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:32.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:33.075 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:32.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:33.075 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:32.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:33.075 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:32.671+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:33.075 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:32.880+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:33.075 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:32.880+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:33.075 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:32.880+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:33.075 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:32.881+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:33.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:33.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:33.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:33.073+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:33.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:33.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:33.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:33.709+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:34.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:34.024+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:34.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:34.024+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:34.024+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:34.025+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:33.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:33.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:33.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:33.862+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:34.848 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:34.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:34.849 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:34.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:34.849 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:34.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:34.849 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:34.699+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:35.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:35.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:35.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:35.028+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:35.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:34.887+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:35.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:34.887+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:35.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:34.887+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:35.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:34.887+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:35.984 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:35.717+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:35.984 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:35.717+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:35.984 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:35.717+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:35.984 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:35.718+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:35.985 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:35.850+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:35.985 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:35.850+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:35.985 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:35.850+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:35.985 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:35.851+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:36.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:35.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:36.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:35.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:35.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:35.982+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:36.759 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.757+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:50:36.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.757+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:50:36.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.757+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f17e8008390 tx=0x7f17e8042170 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:50:36.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.757+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f17e8008390 tx=0x7f17e8042170 comp rx=0 tx=0).stop 2024-09-10T03:50:36.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.757+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:50:36.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.757+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:36.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.757+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:36.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.757+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:36.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.757+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804f5b0 con 0x7f17d8047ae0 2024-09-10T03:50:36.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.757+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:50:36.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.757+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:50:36.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.758+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:50:36.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.758+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:36.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.758+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80485d0 con 0x7f17d8047ae0 2024-09-10T03:50:36.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.758+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f17e001afd0 tx=0x7f17e8004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:50:36.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.758+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028080 con 0x7f17d8047ae0 2024-09-10T03:50:36.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.759+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0018ec0 con 0x7f17d8047ae0 2024-09-10T03:50:36.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:36.760+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d8047ae0 2024-09-10T03:50:36.978 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:50:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:50:36.757+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 2750 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:50:36.978 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:36.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:36.978 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:36.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:36.978 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:36.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:36.978 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:36.699+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:36.978 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:36.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:36.979 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:36.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:36.979 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:36.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:36.979 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:36.876+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:37.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:36.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:37.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:36.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:37.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:36.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:37.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:36.976+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:38.023 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:37.716+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:38.023 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:37.716+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:38.023 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:37.716+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:38.023 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:37.716+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:38.024 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:37.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:38.024 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:37.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:38.024 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:37.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:38.024 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:37.845+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:38.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:38.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:38.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:38.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:38.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:38.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:38.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:38.021+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:39.044 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:38.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:39.045 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:38.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:39.045 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:38.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:39.045 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:38.699+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:39.045 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:38.803+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:39.045 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:38.803+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:39.045 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:38.803+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:39.045 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:38.804+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:39.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:39.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:39.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:39.043+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:40.063 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:39.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:40.063 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:39.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:40.063 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:39.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:40.064 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:39.670+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:40.064 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:39.819+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:40.064 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:39.819+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:40.064 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:39.819+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:40.064 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:39.819+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:40.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:40.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:40.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:40.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:40.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:40.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:40.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:40.062+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:41.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:40.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:41.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:40.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:41.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:40.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:41.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:40.677+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:41.017 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:40.779+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:41.017 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:40.779+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:41.017 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:40.779+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:41.018 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:40.779+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:41.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:41.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:41.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:41.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:41.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:41.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:41.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:41.016+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:41.760 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.758+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:50:41.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.758+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:50:41.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.758+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f17e001afd0 tx=0x7f17e8004970 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:50:41.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.759+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f17e001afd0 tx=0x7f17e8004970 comp rx=0 tx=0).stop 2024-09-10T03:50:41.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.759+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:50:41.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.759+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:41.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.759+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:41.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.759+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:41.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.759+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804f5b0 con 0x7f17d805a5c0 2024-09-10T03:50:41.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.759+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:50:41.761 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.759+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:50:41.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.759+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:50:41.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.759+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:41.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.759+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80487a0 con 0x7f17d805a5c0 2024-09-10T03:50:41.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.760+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f17e803e4d0 tx=0x7f17e0047430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:50:41.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.761+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8015420 con 0x7f17d805a5c0 2024-09-10T03:50:41.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.761+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8029850 con 0x7f17d805a5c0 2024-09-10T03:50:41.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:41.762+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064070 con 0x7f17d805a5c0 2024-09-10T03:50:42.051 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:41.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:42.052 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:41.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:42.052 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:41.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:42.052 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:41.658+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:42.052 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:50:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:50:41.759+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 2814 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:50:42.052 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:41.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:42.052 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:41.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:42.053 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:41.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:42.053 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:41.828+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:42.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:42.050+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:42.050+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:42.050+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:42.051+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:43.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:42.819+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:43.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:42.819+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:43.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:42.819+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:43.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:42.819+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:43.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:42.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:43.058 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:42.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:43.058 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:42.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:43.058 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:42.632+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:43.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:43.055+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:43.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:43.055+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:43.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:43.055+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:43.056+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:43.827 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:43.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:43.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:43.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:43.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:43.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:43.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:43.601+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:44.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:44.064+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:50:44.065 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:44.064+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:50:44.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:43.826+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:44.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:43.826+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:44.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:43.826+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:44.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:43.827+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:44.573 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:44.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:44.573 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:44.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:44.573 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:44.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:44.574 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:44.098+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:44.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:44.572+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:44.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:44.572+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:44.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:44.572+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:44.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:44.573+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:44.874+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:44.874+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:44.874+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:44.875+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:45.093+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:45.093+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:45.093+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:45.094+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:45.850 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:45.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:45.850 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:45.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:45.850 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:45.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:45.850 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:45.591+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:46.128 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:45.849+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:46.128 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:45.849+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:46.128 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:45.849+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:46.128 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:45.849+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:46.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:46.126+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:46.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:46.126+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:46.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:46.126+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:46.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:46.127+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:46.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.760+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:50:46.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.760+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:50:46.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.760+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f17e803e4d0 tx=0x7f17e0047430 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:50:46.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.760+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f17e803e4d0 tx=0x7f17e0047430 comp rx=0 tx=0).stop 2024-09-10T03:50:46.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.760+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:50:46.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.760+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:46.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.760+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:46.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.760+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:46.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.760+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804f5b0 con 0x7f17d8047ae0 2024-09-10T03:50:46.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.760+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:50:46.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.761+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:50:46.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.761+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:50:46.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.761+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:46.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.761+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d4005a90 con 0x7f17d8047ae0 2024-09-10T03:50:46.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.762+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f17e004f230 tx=0x7f17e8017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:50:46.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.762+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055070 con 0x7f17d8047ae0 2024-09-10T03:50:46.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.763+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d8047ae0 2024-09-10T03:50:46.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:46.764+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064070 con 0x7f17d8047ae0 2024-09-10T03:50:47.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:50:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:50:46.760+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 2880 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:50:47.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:46.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:47.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:46.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:46.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:46.633+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:46.838+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:46.838+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:46.838+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:46.839+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:47.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:47.126+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:47.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:47.126+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:47.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:47.126+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:47.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:47.126+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:48.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:47.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:48.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:47.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:48.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:47.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:47.879+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:47.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:47.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:47.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:47.679+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:48.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:48.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:48.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:48.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:48.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:48.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:48.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:48.095+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:49.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:48.635+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:49.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:48.635+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:49.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:48.635+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:48.636+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:48.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:48.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:48.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:48.875+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:49.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:49.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:49.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:49.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:49.128+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:50.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:49.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:50.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:49.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:50.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:49.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:50.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:49.903+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:49.668+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:49.668+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:49.668+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:49.669+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:50.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:50.117+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:50.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:50.117+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:50.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:50.117+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:50.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:50.118+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:51.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:50.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:51.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:50.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:51.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:50.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:51.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:50.855+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:51.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:50.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:51.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:50.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:51.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:50.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:51.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:50.682+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:51.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:51.112+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:51.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:51.112+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:51.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:51.112+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:51.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:51.112+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:51.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.760+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:50:51.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.760+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:50:51.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.761+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f17e004f230 tx=0x7f17e8017040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:50:51.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.761+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f17e004f230 tx=0x7f17e8017040 comp rx=0 tx=0).stop 2024-09-10T03:50:51.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.761+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:50:51.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.761+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:51.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.761+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:51.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.761+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:51.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.761+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804f5b0 con 0x7f17d805a5c0 2024-09-10T03:50:51.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.761+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:50:51.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.761+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:50:51.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.762+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:50:51.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.762+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:51.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.762+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80487a0 con 0x7f17d805a5c0 2024-09-10T03:50:51.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.762+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f17e805f220 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:50:51.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.763+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055040 con 0x7f17d805a5c0 2024-09-10T03:50:51.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.764+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e802ed00 con 0x7f17d805a5c0 2024-09-10T03:50:51.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:51.766+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064070 con 0x7f17d805a5c0 2024-09-10T03:50:52.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:50:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:50:51.761+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 2953 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:50:52.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:51.700+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:52.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:51.700+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:52.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:51.700+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:52.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:51.701+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:51.828+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:51.828+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:51.828+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:51.829+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:52.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:52.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:52.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:52.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:52.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:52.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:52.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:52.150+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:53.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:52.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:53.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:52.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:53.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:52.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:53.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:52.732+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:52.850+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:52.850+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:52.850+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:52.851+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:53.545 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:53.193+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:53.546 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:53.193+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:53.546 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:53.193+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:53.546 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:53.193+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:53.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:53.717+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:53.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:53.717+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:53.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:53.717+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:53.718+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:54.211 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:53.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:54.211 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:53.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:54.211 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:53.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:54.211 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:53.845+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:54.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:54.209+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:54.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:54.209+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:54.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:54.209+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:54.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:54.210+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:55.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:54.706+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:55.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:54.706+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:55.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:54.706+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:55.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:54.706+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:55.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:54.828+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:55.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:54.828+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:55.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:54.828+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:55.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:54.828+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:55.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:55.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:55.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:55.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:55.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:55.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:55.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:55.162+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:56.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:55.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:56.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:55.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:56.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:55.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:55.705+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:55.792+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:55.792+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:55.792+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:55.793+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:56.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:56.154+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:56.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:56.154+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:56.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:56.154+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:56.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:56.155+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:56.762 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.761+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:50:56.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.761+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:50:56.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.761+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f17e805f220 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:50:56.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.761+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f17e805f220 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:50:56.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.761+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:50:56.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.761+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:56.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.761+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:56.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.761+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:50:56.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.761+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804f5b0 con 0x7f17d8047ae0 2024-09-10T03:50:56.763 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.761+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:50:56.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.762+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:50:56.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.762+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:50:56.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.762+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:50:56.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.763+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d4005a90 con 0x7f17d8047ae0 2024-09-10T03:50:56.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.763+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f17d802f250 tx=0x7f17e8017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:50:56.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.763+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0017e20 con 0x7f17d8047ae0 2024-09-10T03:50:56.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.763+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0006be0 con 0x7f17d8047ae0 2024-09-10T03:50:56.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:56.765+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8047ae0 2024-09-10T03:50:57.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:50:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:50:56.761+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 3017 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:50:57.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:56.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:57.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:56.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:57.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:56.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:57.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:56.658+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:56.811+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:56.811+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:56.811+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:56.811+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:57.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:57.143+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:57.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:57.143+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:57.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:57.143+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:57.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:57.143+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:58.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:57.824+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:58.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:57.824+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:57.824+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:57.825+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:57.639+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:57.639+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:57.639+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:57.640+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:58.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:58.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:58.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:58.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:58.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:58.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:58.099+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:59.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:58.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:59.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:58.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:59.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:58.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:58.652+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:50:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:58.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:58.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:58.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:50:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:58.863+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:59.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:59.423+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:50:59.424 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:50:59.423+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:50:59.425 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:59.135+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:59.425 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:59.135+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:59.425 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:59.135+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:50:59.425 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:50:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:50:59.136+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:50:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:59.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:59.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:59.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:50:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:50:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:50:59.634+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:00.174 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:59.876+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:00.174 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:59.876+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:00.174 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:59.876+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:00.174 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:50:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:50:59.877+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:00.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:00.172+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:00.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:00.172+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:00.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:00.172+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:00.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:00.173+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:01.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:00.859+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:01.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:00.859+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:00.859+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:00.859+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:01.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:00.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:01.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:00.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:01.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:00.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:01.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:00.614+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:01.575 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:01.193+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:01.575 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:01.193+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:01.575 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:01.193+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:01.575 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:01.194+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:01.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.762+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:51:01.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.762+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:51:01.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.762+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f17d802f250 tx=0x7f17e8017040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:51:01.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.762+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f17d802f250 tx=0x7f17e8017040 comp rx=0 tx=0).stop 2024-09-10T03:51:01.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.762+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:51:01.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.762+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:01.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.762+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:01.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.762+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:01.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.762+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804f5b0 con 0x7f17d805a5c0 2024-09-10T03:51:01.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.762+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:51:01.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.762+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:51:01.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.763+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:51:01.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.763+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:01.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.763+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80487a0 con 0x7f17d805a5c0 2024-09-10T03:51:01.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.764+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f17e8038f90 tx=0x7f17e0009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:51:01.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.764+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e800d6e0 con 0x7f17d805a5c0 2024-09-10T03:51:01.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.765+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e80093a0 con 0x7f17d805a5c0 2024-09-10T03:51:01.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:01.766+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064050 con 0x7f17d805a5c0 2024-09-10T03:51:01.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:51:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:51:01.762+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 3086 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:51:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:01.573+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:01.573+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:01.573+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:01.574+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:01.812+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:01.812+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:01.812+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:01.813+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:02.497 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:02.209+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:02.497 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:02.209+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:02.497 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:02.209+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:02.497 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:02.210+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:03.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:02.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:03.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:02.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:03.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:02.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:03.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:02.601+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:02.837+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:02.837+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:02.837+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:02.838+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:03.546 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:03.178+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:03.546 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:03.178+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:03.546 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:03.178+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:03.547 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:03.179+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:03.575+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:03.575+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:03.575+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:03.575+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:04.150 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:03.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:04.150 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:03.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:04.150 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:03.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:04.151 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:03.882+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:04.550 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:04.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:04.550 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:04.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:04.550 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:04.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:04.550 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:04.149+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:04.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:04.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:04.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:04.549+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:05.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:05.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:05.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:05.150+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:04.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:04.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:04.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:04.930+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:05.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:05.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:05.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:05.585+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:06.198 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:05.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:06.198 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:05.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:06.198 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:05.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:06.198 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:05.898+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:06.565 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:06.196+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:06.565 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:06.196+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:06.565 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:06.196+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:06.565 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:06.197+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:06.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.762+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:51:06.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.763+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:51:06.764 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.763+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f17e8038f90 tx=0x7f17e0009190 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:51:06.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.763+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f17e8038f90 tx=0x7f17e0009190 comp rx=0 tx=0).stop 2024-09-10T03:51:06.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.763+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:51:06.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.763+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:06.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.763+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:06.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.763+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:06.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.763+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059da0 con 0x7f17d8047ae0 2024-09-10T03:51:06.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.763+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048970 con 0x7f17f0101a40 2024-09-10T03:51:06.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.763+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:51:06.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.764+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:51:06.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.764+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:06.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.764+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048af0 con 0x7f17d8047ae0 2024-09-10T03:51:06.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.764+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f17d8017880 tx=0x7f17e8017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:51:06.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.764+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e80045b0 con 0x7f17d8047ae0 2024-09-10T03:51:06.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.765+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8035be0 con 0x7f17d8047ae0 2024-09-10T03:51:06.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:06.766+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064050 con 0x7f17d8047ae0 2024-09-10T03:51:06.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:51:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:51:06.763+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 3151 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:51:06.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:06.564+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:06.564+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:06.564+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:06.565+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:07.223 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:06.860+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:07.223 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:06.860+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:07.223 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:06.860+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:07.223 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:06.861+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:07.521 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:07.221+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:07.521 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:07.221+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:07.521 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:07.221+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:07.521 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:07.222+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:07.819 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:07.519+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:07.819 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:07.519+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:07.819 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:07.519+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:07.819 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:07.520+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:07.817+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:07.817+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:07.817+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:08.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:07.818+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:08.524 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:08.259+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:08.524 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:08.259+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:08.524 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:08.259+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:08.524 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:08.260+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:08.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:08.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:08.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:08.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:08.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:08.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:08.523+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:09.294 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:08.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:09.295 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:08.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:09.295 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:08.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:09.295 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:08.864+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:09.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:09.292+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:09.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:09.292+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:09.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:09.292+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:09.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:09.293+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:09.584 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:09.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:09.584 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:09.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:09.584 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:09.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:09.584 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:09.517+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:10.283 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:09.866+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:10.283 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:09.866+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:10.283 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:09.866+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:10.284 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:09.867+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:10.561 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:10.280+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:10.561 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:10.280+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:10.561 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:10.280+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:10.561 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:10.281+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:10.562 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:10.474+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:10.562 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:10.474+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:10.562 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:10.474+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:10.562 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:10.474+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:11.280 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:10.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:11.280 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:10.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:11.280 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:10.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:11.280 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:10.883+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:11.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:11.278+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:11.278+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:11.278+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:11.278+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:11.426+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:11.426+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:11.426+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:11.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:11.427+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:11.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.762+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:51:11.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.762+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:51:11.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.762+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f17d8017880 tx=0x7f17e8017040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:51:11.765 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.762+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f17d8017880 tx=0x7f17e8017040 comp rx=0 tx=0).stop 2024-09-10T03:51:11.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.762+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:51:11.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.762+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:11.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.762+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:11.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.762+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:11.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.762+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d805a5c0 2024-09-10T03:51:11.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.762+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:51:11.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.763+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:51:11.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.763+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:51:11.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.763+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:11.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.763+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80487a0 con 0x7f17d805a5c0 2024-09-10T03:51:11.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.763+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f17e803cff0 tx=0x7f17e8044ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:51:11.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.764+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e800d6e0 con 0x7f17d805a5c0 2024-09-10T03:51:11.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.764+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e80093a0 con 0x7f17d805a5c0 2024-09-10T03:51:11.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:11.766+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064050 con 0x7f17d805a5c0 2024-09-10T03:51:12.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:51:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:51:11.763+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 3216 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:51:12.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:11.842+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:11.842+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:11.842+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:11.842+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:12.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:12.320+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:12.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:12.320+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:12.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:12.320+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:12.321+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:12.390+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:12.390+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:12.390+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:12.391+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:13.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:12.803+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:13.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:12.803+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:13.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:12.803+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:13.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:12.804+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:13.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:13.297+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:13.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:13.297+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:13.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:13.297+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:13.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:13.298+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:13.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:13.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:13.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:13.399+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:13.787+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:13.787+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:13.787+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:13.787+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:14.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:14.326+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:14.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:14.326+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:14.326+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:14.326+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:14.354+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:14.354+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:14.355+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:14.355+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:14.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:14.782+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:51:14.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:14.782+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:51:15.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:14.760+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:15.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:14.760+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:15.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:14.760+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:15.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:14.760+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:15.368 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:15.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:15.368 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:15.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:15.369 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:15.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:15.369 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:15.311+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:15.648 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:15.366+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:15.648 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:15.366+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:15.648 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:15.366+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:15.648 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:15.367+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:16.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:15.750+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:16.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:15.750+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:16.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:15.750+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:16.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:15.751+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:16.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:16.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:16.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:16.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:16.390+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:16.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:16.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:16.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:16.313+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:16.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.764+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:51:16.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.764+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:51:16.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.764+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f17e803cff0 tx=0x7f17e8044ea0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:51:16.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.764+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f17e803cff0 tx=0x7f17e8044ea0 comp rx=0 tx=0).stop 2024-09-10T03:51:16.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.764+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:51:16.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.764+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:16.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.764+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:16.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.764+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:16.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.764+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059da0 con 0x7f17d8047ae0 2024-09-10T03:51:16.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.764+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048970 con 0x7f17f0101a40 2024-09-10T03:51:16.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.765+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:51:16.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.765+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:51:16.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.765+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:16.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.765+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80487a0 con 0x7f17d8047ae0 2024-09-10T03:51:16.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.766+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f17d8017880 tx=0x7f17e8017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:51:16.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.767+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028070 con 0x7f17d8047ae0 2024-09-10T03:51:16.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.767+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e001fe80 con 0x7f17d8047ae0 2024-09-10T03:51:16.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:16.769+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8047ae0 2024-09-10T03:51:17.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:51:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:51:16.764+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 3280 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:51:17.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:16.727+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:16.727+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:16.727+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:16.728+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:17.705 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:17.397+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:17.706 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:17.397+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:17.706 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:17.397+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:17.706 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:17.397+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:17.706 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:17.348+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:17.706 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:17.348+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:17.706 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:17.348+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:17.706 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:17.349+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:18.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:17.703+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:18.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:17.703+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:18.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:17.703+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:18.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:17.705+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:18.707 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:18.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:18.708 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:18.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:18.708 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:18.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:18.708 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:18.384+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:18.708 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:18.395+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:18.708 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:18.395+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:18.708 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:18.395+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:18.708 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:18.395+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:19.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:18.706+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:19.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:18.706+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:19.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:18.706+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:19.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:18.707+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:19.743 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:19.365+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:19.744 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:19.365+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:19.744 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:19.365+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:19.744 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:19.365+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:19.745 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:19.392+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:19.745 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:19.392+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:19.745 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:19.392+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:19.745 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:19.392+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:20.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:19.742+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:20.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:19.742+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:20.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:19.742+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:20.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:19.742+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:20.714 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:20.389+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:20.714 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:20.389+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:20.714 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:20.389+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:20.714 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:20.390+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:20.715 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:20.391+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:20.715 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:20.391+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:20.715 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:20.391+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:20.715 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:20.392+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:21.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:20.713+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:21.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:20.713+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:21.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:20.713+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:20.713+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:21.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:21.393+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:21.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:21.393+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:21.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:21.393+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:21.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:21.394+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:21.756 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:21.436+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:21.757 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:21.436+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:21.757 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:21.436+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:21.757 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:21.437+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:21.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:51:21.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:51:21.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f17d8017880 tx=0x7f17e8017040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:51:21.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f17d8017880 tx=0x7f17e8017040 comp rx=0 tx=0).stop 2024-09-10T03:51:21.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:51:21.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:21.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:21.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:21.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059da0 con 0x7f17d805a5c0 2024-09-10T03:51:21.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80487a0 con 0x7f17f0101a40 2024-09-10T03:51:21.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:51:21.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:51:21.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:21.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048920 con 0x7f17d805a5c0 2024-09-10T03:51:21.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f17e8040ce0 tx=0x7f17e0059b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:51:21.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e803f400 con 0x7f17d805a5c0 2024-09-10T03:51:21.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e80093a0 con 0x7f17d805a5c0 2024-09-10T03:51:21.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:21.765+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064050 con 0x7f17d805a5c0 2024-09-10T03:51:22.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:51:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:51:21.765+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 3351 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:51:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:21.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:21.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:21.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:21.755+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:22.784 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:22.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:22.785 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:22.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:22.785 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:22.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:22.785 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:22.408+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:22.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:22.422+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:22.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:22.422+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:22.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:22.422+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:22.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:22.423+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:23.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:22.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:22.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:22.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:22.783+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:23.764 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:23.401+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:23.764 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:23.401+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:23.764 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:23.401+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:23.764 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:23.401+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:23.764 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:23.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:23.764 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:23.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:23.765 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:23.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:23.765 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:23.420+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:24.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:23.762+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:24.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:23.762+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:24.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:23.762+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:24.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:23.763+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:24.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:24.377+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:24.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:24.377+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:24.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:24.377+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:24.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:24.378+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:24.723 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:24.388+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:24.723 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:24.388+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:24.723 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:24.388+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:24.724 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:24.389+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:25.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:24.721+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:25.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:24.721+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:25.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:24.721+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:25.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:24.722+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:25.769 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:25.372+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:25.769 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:25.372+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:25.769 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:25.372+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:25.770 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:25.372+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:25.770 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:25.402+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:25.770 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:25.402+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:25.770 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:25.402+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:25.770 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:25.403+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:26.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:25.768+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:26.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:25.768+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:26.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:25.768+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:26.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:25.769+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:26.738 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:26.375+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:26.738 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:26.375+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:26.738 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:26.375+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:26.739 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:26.376+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:26.739 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:26.379+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:26.739 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:26.379+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:26.739 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:26.379+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:26.739 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:26.380+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:26.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.765+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:51:26.766 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.765+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:51:26.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.765+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f17e8040ce0 tx=0x7f17e0059b60 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:51:26.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.765+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f17e8040ce0 tx=0x7f17e0059b60 comp rx=0 tx=0).stop 2024-09-10T03:51:26.767 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.765+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:51:26.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.765+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:26.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.765+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:26.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.765+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:26.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.765+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804abc0 con 0x7f17d8047ae0 2024-09-10T03:51:26.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.765+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17f0101a40 2024-09-10T03:51:26.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.766+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:51:26.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.766+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:51:26.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.766+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:26.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.767+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048920 con 0x7f17d8047ae0 2024-09-10T03:51:26.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.767+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f17e0065000 tx=0x7f17e8015f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:51:26.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.769+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c050 con 0x7f17d8047ae0 2024-09-10T03:51:26.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.770+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e00184d0 con 0x7f17d8047ae0 2024-09-10T03:51:26.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:26.770+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8047ae0 2024-09-10T03:51:27.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:51:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:51:26.766+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 3431 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:51:27.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:26.737+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:26.737+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:26.737+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:26.738+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:27.785 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:27.344+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:27.785 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:27.344+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:27.785 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:27.344+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:27.785 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:27.345+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:27.786 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:27.382+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:27.786 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:27.382+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:27.786 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:27.382+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:27.786 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:27.382+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:28.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:27.784+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:28.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:27.784+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:28.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:27.784+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:27.784+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:28.809 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:28.344+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:28.809 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:28.344+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:28.809 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:28.344+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:28.809 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:28.345+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:28.810 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:28.334+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:28.810 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:28.334+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:28.810 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:28.334+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:28.810 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:28.335+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:29.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:28.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:28.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:28.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:28.808+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:29.819 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:29.346+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:29.819 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:29.346+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:29.819 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:29.346+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:29.819 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:29.347+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:29.819 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:29.329+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:29.819 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:29.329+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:29.819 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:29.329+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:29.819 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:29.330+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:30.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:29.817+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:30.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:29.817+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:30.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:29.817+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:29.818+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:30.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:30.335+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:51:30.336 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:30.335+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:51:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:30.364+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:30.364+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:30.364+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:30.365+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:30.329+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:30.329+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:30.329+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:30.329+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:31.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:30.813+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:30.813+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:30.813+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:30.814+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:31.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.766+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:51:31.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.767+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:51:31.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.767+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f17e0065000 tx=0x7f17e8015f00 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:51:31.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.767+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f17e0065000 tx=0x7f17e8015f00 comp rx=0 tx=0).stop 2024-09-10T03:51:31.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.767+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:51:31.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.767+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:31.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.767+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:31.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.767+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:31.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.767+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059d10 con 0x7f17d805a5c0 2024-09-10T03:51:31.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.767+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e90 con 0x7f17f0101a40 2024-09-10T03:51:31.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:31.349+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:31.770 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:31.349+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:31.770 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:31.349+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:31.770 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:31.350+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:31.770 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:31.341+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:31.770 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:31.341+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:31.770 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:31.341+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:31.770 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:31.341+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:31.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.768+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:51:31.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.769+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:51:31.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.769+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:31.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.769+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048880 con 0x7f17d805a5c0 2024-09-10T03:51:31.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.769+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f17e0017f20 tx=0x7f17e002f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:51:31.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.770+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8009c40 con 0x7f17d805a5c0 2024-09-10T03:51:31.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.771+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d805a5c0 2024-09-10T03:51:31.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:31.772+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064070 con 0x7f17d805a5c0 2024-09-10T03:51:32.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:51:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:51:31.766+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 3505 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:51:32.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:31.846+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:32.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:31.846+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:31.846+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:31.847+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:32.372+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:32.372+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:32.372+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:32.373+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:32.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:32.337+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:32.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:32.337+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:32.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:32.337+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:32.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:32.337+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:32.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:32.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:32.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:32.896+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:33.362+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:33.362+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:33.362+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:33.363+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:33.386+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:33.386+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:33.386+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:33.387+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:34.324 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:33.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:34.325 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:33.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:34.325 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:33.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:34.325 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:33.883+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:34.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:34.322+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:34.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:34.322+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:34.322+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:34.323+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:34.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:34.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:34.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:34.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:34.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:34.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:34.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:34.395+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:34.924+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:35.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:34.924+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:35.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:34.924+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:35.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:34.925+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:35.406+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:35.406+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:35.406+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:35.407+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:35.359+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:35.359+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:35.359+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:35.360+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:35.884+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:35.884+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:35.884+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:35.885+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:36.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.767+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:51:36.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.767+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:51:36.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.767+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f17e0017f20 tx=0x7f17e002f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:51:36.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.767+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f17e0017f20 tx=0x7f17e002f040 comp rx=0 tx=0).stop 2024-09-10T03:51:36.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.767+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:51:36.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.767+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:36.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.767+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:36.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.767+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:36.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.767+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059da0 con 0x7f17d8047ae0 2024-09-10T03:51:36.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.767+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048a50 con 0x7f17f0101a40 2024-09-10T03:51:36.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:36.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:36.770 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:36.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:36.770 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:36.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:36.770 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:36.423+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:36.770 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:36.406+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:36.770 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:36.406+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:36.770 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:36.406+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:36.770 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:36.407+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:36.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.767+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:51:36.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.769+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:51:36.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.769+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:36.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.770+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048880 con 0x7f17d8047ae0 2024-09-10T03:51:36.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.771+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f17e002c050 tx=0x7f17e800b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:51:36.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.772+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028050 con 0x7f17d8047ae0 2024-09-10T03:51:36.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.772+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e005d890 con 0x7f17d8047ae0 2024-09-10T03:51:36.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:36.773+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d8047ae0 2024-09-10T03:51:37.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:51:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:51:36.767+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 3569 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:51:37.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:36.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:36.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:36.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:36.883+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:37.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:37.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:37.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:37.409+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:37.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:37.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:37.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:37.415+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:37.929+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:37.929+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:37.929+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:38.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:37.930+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:38.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:38.366+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:38.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:38.366+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:38.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:38.366+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:38.367+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:38.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:38.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:38.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:38.368+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:39.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:38.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:39.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:38.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:39.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:38.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:39.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:38.931+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:39.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:39.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:39.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:39.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:39.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:39.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:39.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:39.385+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:39.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:39.320+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:39.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:39.320+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:39.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:39.320+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:39.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:39.321+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:40.303 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:39.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:40.303 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:39.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:40.303 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:39.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:40.303 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:39.956+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:40.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:40.431+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:40.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:40.431+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:40.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:40.431+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:40.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:40.431+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:40.301+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:40.301+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:40.301+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:40.302+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:41.277 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:40.993+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:41.280 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:40.993+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:41.280 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:40.993+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:41.280 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:40.994+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:41.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:41.276+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:41.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:41.276+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:41.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:41.276+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:41.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:41.277+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:41.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:41.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:41.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:41.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:41.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:41.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:41.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:41.398+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:41.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.768+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:51:41.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.768+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:51:41.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.768+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f17e002c050 tx=0x7f17e800b6e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:51:41.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.768+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f17e002c050 tx=0x7f17e800b6e0 comp rx=0 tx=0).stop 2024-09-10T03:51:41.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.768+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:51:41.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.768+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:41.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.768+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:41.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.768+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:41.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.768+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059da0 con 0x7f17d805a5c0 2024-09-10T03:51:41.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.768+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048880 con 0x7f17f0101a40 2024-09-10T03:51:41.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.768+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:51:41.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.769+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:51:41.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.769+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:41.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.769+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048a00 con 0x7f17d805a5c0 2024-09-10T03:51:41.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.769+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f17e00104d0 tx=0x7f17e002dc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:51:41.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.770+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c040 con 0x7f17d805a5c0 2024-09-10T03:51:41.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.770+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0015d70 con 0x7f17d805a5c0 2024-09-10T03:51:41.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:41.772+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d805a5c0 2024-09-10T03:51:42.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:51:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:51:41.768+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 3633 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:51:42.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:41.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:42.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:41.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:42.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:41.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:42.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:41.958+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:42.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:42.438+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:42.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:42.438+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:42.438+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:42.439+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:42.262+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:42.262+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:42.262+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:42.263+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:43.272 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:42.949+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:43.273 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:42.949+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:43.273 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:42.949+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:43.273 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:42.950+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:43.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:43.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:43.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:43.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:43.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:43.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:43.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:43.413+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:43.550 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:43.271+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:43.550 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:43.271+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:43.550 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:43.271+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:43.550 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:43.272+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:44.245 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:43.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:44.246 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:43.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:44.246 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:43.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:44.246 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:43.909+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:44.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:44.244+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:44.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:44.244+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:44.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:44.244+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:44.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:44.245+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:44.428+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:44.428+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:44.428+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:44.429+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:45.225 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:44.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:45.225 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:44.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:45.225 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:44.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:45.225 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:44.941+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:45.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:45.223+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:45.223+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:45.223+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:45.224+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:45.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:45.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:45.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:45.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:45.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:45.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:45.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:45.438+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:45.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:45.695+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:51:45.696 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:45.695+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:51:46.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:45.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:46.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:45.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:46.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:45.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:46.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:45.929+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:46.463 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:46.179+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:46.463 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:46.179+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:46.463 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:46.179+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:46.464 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:46.180+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:46.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.767+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:51:46.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.767+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:51:46.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.767+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f17e00104d0 tx=0x7f17e002dc20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:51:46.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.767+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f17e00104d0 tx=0x7f17e002dc20 comp rx=0 tx=0).stop 2024-09-10T03:51:46.770 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.768+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:51:46.771 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:46.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:46.771 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:46.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:46.771 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:46.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:46.771 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:46.461+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:46.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.768+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:46.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.768+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:46.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.768+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:46.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.768+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8059e80 con 0x7f17d8047ae0 2024-09-10T03:51:46.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.768+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80487a0 con 0x7f17f0101a40 2024-09-10T03:51:46.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.768+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:51:46.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.769+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:51:46.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.769+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:46.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.769+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048920 con 0x7f17d8047ae0 2024-09-10T03:51:46.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.769+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f17e0030f90 tx=0x7f17e002f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:51:46.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.770+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028040 con 0x7f17d8047ae0 2024-09-10T03:51:46.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.770+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0015d70 con 0x7f17d8047ae0 2024-09-10T03:51:46.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:46.771+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8047ae0 2024-09-10T03:51:47.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:51:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:51:46.768+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 3696 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:51:47.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:46.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:46.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:46.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:46.931+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:47.417 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:47.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:47.417 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:47.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:47.417 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:47.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:47.418 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:47.157+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:47.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:47.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:47.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:47.415+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:48.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:48.190+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:48.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:48.191+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:48.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:48.191+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:48.191+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:47.969+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:47.969+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:47.969+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:47.970+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:48.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:48.439+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:48.925 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:48.439+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:48.925 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:48.439+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:48.925 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:48.440+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:49.182 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:48.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:49.183 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:48.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:49.183 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:48.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:49.183 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:48.926+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:49.465 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:49.179+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:49.465 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:49.179+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:49.465 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:49.179+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:49.465 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:49.180+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:49.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:49.462+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:49.462+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:49.462+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:49.463+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:50.221 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:49.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:50.221 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:49.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:50.221 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:49.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:50.221 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:49.906+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:50.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:50.219+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:50.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:50.219+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:50.219+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:50.219+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:50.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:50.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:50.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:50.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:50.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:50.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:50.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:50.464+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:51.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:50.861+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:51.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:50.861+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:51.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:50.861+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:51.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:50.862+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:51.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:51.254+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:51.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:51.254+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:51.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:51.254+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:51.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:51.254+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:51.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:51.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:51.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:51.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:51.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:51.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:51.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:51.447+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:51.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.769+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:51:51.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.769+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:51:51.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.769+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f17e0030f90 tx=0x7f17e002f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:51:51.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.769+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f17e0030f90 tx=0x7f17e002f040 comp rx=0 tx=0).stop 2024-09-10T03:51:51.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.769+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:51:51.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.769+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:51.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.769+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:51.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.769+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:51.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.769+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80487a0 con 0x7f17d805a5c0 2024-09-10T03:51:51.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.769+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:51:51.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.770+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:51:51.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.770+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:51:51.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.770+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:51.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.770+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80485d0 con 0x7f17d805a5c0 2024-09-10T03:51:51.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.771+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f17d8011cf0 tx=0x7f17e0050fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:51:51.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.771+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055040 con 0x7f17d805a5c0 2024-09-10T03:51:51.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.771+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8042e90 con 0x7f17d805a5c0 2024-09-10T03:51:51.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:51.773+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064070 con 0x7f17d805a5c0 2024-09-10T03:51:52.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:51:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:51:51.769+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 3759 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:51:52.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:51.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:51.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:51.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:51.882+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:52.477 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:52.298+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:52.478 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:52.298+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:52.478 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:52.298+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:52.478 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:52.299+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:52.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:52.475+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:52.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:52.475+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:52.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:52.475+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:52.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:52.477+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:52.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:52.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:52.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:52.895+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:53.336+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:53.336+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:53.336+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:53.337+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:53.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:53.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:53.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:53.461+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:54.307 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:53.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:54.307 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:53.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:54.307 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:53.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:54.307 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:53.925+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:54.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:54.481+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:54.481+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:54.481+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:54.482+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:54.305+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:54.305+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:54.305+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:54.306+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:54.962+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:54.962+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:54.962+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:54.962+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:55.333+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:55.333+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:55.333+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:55.334+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:55.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:55.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:55.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:55.510+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:55.962+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:55.962+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:55.962+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:55.962+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:56.771 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.770+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:51:56.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.770+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:51:56.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.770+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f17d8011cf0 tx=0x7f17e0050fc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:51:56.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.770+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f17d8011cf0 tx=0x7f17e0050fc0 comp rx=0 tx=0).stop 2024-09-10T03:51:56.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.770+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:51:56.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.770+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:56.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.770+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:56.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.770+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:51:56.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.770+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80487a0 con 0x7f17d8047ae0 2024-09-10T03:51:56.772 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.770+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:51:56.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:56.520+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:56.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:56.520+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:56.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:56.520+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:56.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:56.520+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:56.773 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:56.358+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:56.773 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:56.358+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:56.773 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:56.358+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:56.773 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:56.359+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:56.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.771+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:51:56.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.771+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:51:56.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.771+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:51:56.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.771+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048aa0 con 0x7f17d8047ae0 2024-09-10T03:51:56.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.772+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f17d800fce0 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:51:56.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.772+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8026680 con 0x7f17d8047ae0 2024-09-10T03:51:56.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.772+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d8047ae0 2024-09-10T03:51:56.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:51:56.774+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c040 con 0x7f17d8047ae0 2024-09-10T03:51:57.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:51:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:51:56.771+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 3822 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:51:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:57.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:57.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:57.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:57.003+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:57.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:57.355+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:57.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:57.355+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:57.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:57.355+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:57.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:57.356+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:57.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:57.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:57.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:57.542+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:58.329 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:58.053+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:58.329 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:58.053+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:58.329 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:58.053+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:58.054+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:58.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:58.328+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:58.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:58.328+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:58.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:58.328+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:58.329+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:58.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:58.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:58.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:58.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:58.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:58.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:58.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:58.513+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:51:59.488 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:59.101+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:59.489 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:59.101+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:59.489 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:59.101+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:51:59.489 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:51:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:51:59.102+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:59.489 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:59.307+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:59.490 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:59.307+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:59.490 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:59.307+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:51:59.490 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:51:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:51:59.308+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:51:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:59.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:59.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:59.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:51:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:51:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:51:59.487+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:00.357 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:00.102+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:00.357 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:00.102+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:00.357 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:00.102+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:00.357 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:00.103+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:00.356+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:00.356+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:00.356+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:00.356+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:00.508+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:00.508+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:00.508+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:00.509+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:01.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.055+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:52:01.056 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.055+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:52:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:01.069+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:01.069+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:01.069+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:01.069+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:01.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.771+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:52:01.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.771+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:52:01.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.771+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f17d800fce0 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:52:01.773 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.771+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f17d800fce0 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:52:01.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.771+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:52:01.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.771+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:01.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.771+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:01.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.772+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:01.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.772+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80487a0 con 0x7f17d805a5c0 2024-09-10T03:52:01.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.772+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:52:01.774 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:01.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:01.774 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:01.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:01.774 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:01.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:01.775 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:01.527+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:01.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.772+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:52:01.775 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:01.404+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:01.775 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:01.404+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:01.775 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:01.404+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:01.775 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:01.405+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:01.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.772+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:52:01.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.772+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:01.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.773+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80485d0 con 0x7f17d805a5c0 2024-09-10T03:52:01.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.773+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f17e803e3d0 tx=0x7f17e8026940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:52:01.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.774+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e800d9c0 con 0x7f17d805a5c0 2024-09-10T03:52:01.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.774+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803f840 con 0x7f17d805a5c0 2024-09-10T03:52:01.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:01.775+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c040 con 0x7f17d805a5c0 2024-09-10T03:52:02.023 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:52:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:52:01.772+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 3885 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:52:02.024 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:02.021+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:02.024 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:02.021+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:02.024 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:02.021+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:02.024 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:02.022+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:02.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:02.391+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:02.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:02.391+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:02.391+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:02.392+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:02.541+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:02.541+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:02.541+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:02.542+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:03.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:03.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:03.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:03.060+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:03.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:03.401+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:03.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:03.401+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:03.401+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:03.401+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:03.543+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:03.543+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:03.543+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:03.544+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:04.066+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:04.066+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:04.066+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:04.066+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:04.440+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:04.440+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:04.440+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:04.441+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:04.494+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:04.494+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:04.494+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:04.495+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:05.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:05.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:05.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:05.035+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:05.393+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:05.393+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:05.393+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:05.394+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:05.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:05.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:05.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:05.507+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:05.993+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:05.993+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:05.993+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:05.994+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:06.671 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:06.370+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:06.671 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:06.370+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:06.671 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:06.370+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:06.671 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:06.370+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:06.672 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:06.474+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:06.672 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:06.474+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:06.672 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:06.474+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:06.672 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:06.475+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:06.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.772+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:52:06.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.772+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:52:06.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.772+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f17e803e3d0 tx=0x7f17e8026940 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:52:06.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.772+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f17e803e3d0 tx=0x7f17e8026940 comp rx=0 tx=0).stop 2024-09-10T03:52:06.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.772+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:52:06.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.772+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:06.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.772+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:06.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.772+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:06.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.772+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80487a0 con 0x7f17d8047ae0 2024-09-10T03:52:06.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.772+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:52:06.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.773+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:52:06.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.773+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:52:06.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.773+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:06.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.773+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048aa0 con 0x7f17d8047ae0 2024-09-10T03:52:06.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.774+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f17d802cbb0 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:52:06.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.775+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028070 con 0x7f17d8047ae0 2024-09-10T03:52:06.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.775+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e001bca0 con 0x7f17d8047ae0 2024-09-10T03:52:06.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:06.777+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8047ae0 2024-09-10T03:52:07.019 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:52:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:52:06.772+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 3948 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:52:07.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:07.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:07.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:07.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:07.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:07.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:07.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:07.019+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:07.392+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:07.392+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:07.392+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:07.392+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:07.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:07.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:07.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:07.480+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:08.033+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:08.033+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:08.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:08.033+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:08.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:08.034+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:08.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:08.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:08.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:08.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:08.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:08.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:08.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:08.402+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:08.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:08.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:08.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:08.494+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:09.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:09.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:09.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:09.008+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:09.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:09.419+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:09.419+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:09.419+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:09.420+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:09.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:09.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:09.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:09.464+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:10.017+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:10.017+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:10.017+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:10.018+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:10.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:10.440+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:10.440+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:10.440+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:10.441+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:10.452+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:10.452+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:10.452+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:10.453+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:10.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:10.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:10.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:11.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:10.971+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:11.765 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:11.484+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:11.765 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:11.484+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:11.765 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:11.484+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:11.765 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:11.485+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:11.765 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:11.411+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:11.765 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:11.411+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:11.765 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:11.411+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:11.766 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:11.411+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.773+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:52:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.773+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:52:11.774 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.773+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f17d802cbb0 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:52:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.773+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f17d802cbb0 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:52:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.773+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:52:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.773+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.773+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.773+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.773+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80487a0 con 0x7f17d805a5c0 2024-09-10T03:52:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.773+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:52:11.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.773+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:52:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.774+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:52:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.774+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.774+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048c70 con 0x7f17d805a5c0 2024-09-10T03:52:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.774+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f17e8008550 tx=0x7f17e002dc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:52:11.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.775+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055070 con 0x7f17d805a5c0 2024-09-10T03:52:11.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.775+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d805a5c0 2024-09-10T03:52:11.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:11.776+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17d805a5c0 2024-09-10T03:52:12.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:52:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:52:11.773+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 4011 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:52:12.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:11.963+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:12.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:11.963+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:11.963+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:11.964+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:12.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:12.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:12.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:12.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:12.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:12.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:12.454+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:12.456+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:12.456+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:12.456+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:12.457+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:13.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:12.969+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:13.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:12.969+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:13.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:12.969+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:13.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:12.970+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:13.413+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:13.413+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:13.413+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:13.414+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:13.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:13.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:13.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:13.414+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:13.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:13.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:13.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:13.943+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:14.408+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:14.408+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:14.408+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:14.409+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:14.419+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:14.419+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:14.419+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:14.419+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:14.947+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:14.947+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:14.947+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:14.948+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:15.402+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:15.402+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:15.402+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:15.403+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:15.448+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:15.448+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:15.448+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:15.449+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:16.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:15.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:16.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:15.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:16.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:15.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:16.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:15.926+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:16.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.479+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:52:16.480 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.479+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:52:16.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.773+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:52:16.775 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.773+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:52:16.775 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:16.489+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:16.775 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:16.489+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:16.776 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:16.489+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:16.776 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:16.490+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:16.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:16.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:16.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:16.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:16.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:16.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:16.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:16.417+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:16.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.774+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f17e8008550 tx=0x7f17e002dc20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:52:16.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.774+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f17e8008550 tx=0x7f17e002dc20 comp rx=0 tx=0).stop 2024-09-10T03:52:16.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.774+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:52:16.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.774+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:16.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.774+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:16.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.774+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:16.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.774+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80487a0 con 0x7f17d8047ae0 2024-09-10T03:52:16.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.774+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:52:16.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.775+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:52:16.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.775+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:52:16.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.775+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:16.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.775+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048c70 con 0x7f17d8047ae0 2024-09-10T03:52:16.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.776+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f17d8011170 tx=0x7f17e800b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:52:16.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.777+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c040 con 0x7f17d8047ae0 2024-09-10T03:52:16.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.779+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e000ee50 con 0x7f17d8047ae0 2024-09-10T03:52:16.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:16.779+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8047ae0 2024-09-10T03:52:17.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:52:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:52:16.774+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 4074 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:52:17.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:16.926+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:16.926+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:16.926+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:16.927+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:17.525+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:17.525+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:17.525+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:17.526+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:17.402+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:17.402+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:17.402+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:17.403+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:17.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:17.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:17.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:18.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:17.956+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:18.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:18.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:18.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:18.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:18.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:18.399+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:18.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:18.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:18.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:18.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:18.493+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:19.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:18.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:19.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:18.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:19.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:18.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:19.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:18.909+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:19.483+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:19.483+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:19.483+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:19.484+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:19.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:19.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:19.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:19.378+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:20.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:19.934+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:20.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:19.934+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:19.934+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:19.935+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:20.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:20.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:20.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:20.452+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:20.356+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:20.356+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:20.356+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:20.356+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:20.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:20.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:20.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:20.918+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:21.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.774+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:52:21.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.774+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:52:21.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.774+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f17d8011170 tx=0x7f17e800b6e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:52:21.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.774+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f17d8011170 tx=0x7f17e800b6e0 comp rx=0 tx=0).stop 2024-09-10T03:52:21.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.774+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:52:21.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.774+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:21.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.774+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:21.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.774+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:21.776 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.774+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80487a0 con 0x7f17d805a5c0 2024-09-10T03:52:21.777 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.774+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:52:21.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:21.370+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:21.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:21.370+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:21.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:21.370+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:21.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:21.371+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:21.778 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:21.431+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:21.778 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:21.431+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:21.778 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:21.431+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:21.778 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:21.432+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:21.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.776+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:52:21.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.776+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:52:21.778 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.776+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:21.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.776+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048c70 con 0x7f17d805a5c0 2024-09-10T03:52:21.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.777+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f17e801afc0 tx=0x7f17e005d670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:52:21.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.778+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055070 con 0x7f17d805a5c0 2024-09-10T03:52:21.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.778+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e804ec90 con 0x7f17d805a5c0 2024-09-10T03:52:21.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:21.779+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17d805a5c0 2024-09-10T03:52:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:21.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:21.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:21.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:21.902+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:22.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:52:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:52:21.775+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 4137 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:52:22.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:22.383+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:22.383+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:22.383+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:22.384+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:22.409+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:22.409+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:22.409+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:22.410+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:23.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:22.896+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:23.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:22.896+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:23.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:22.896+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:22.897+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:23.417+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:23.417+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:23.417+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:23.418+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:23.393+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:23.393+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:23.393+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:23.393+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:23.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:23.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:23.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:23.928+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:24.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:24.359+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:24.359+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:24.359+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:24.359+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:24.398+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:24.398+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:24.398+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:24.399+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:25.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:24.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:25.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:24.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:25.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:24.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:25.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:24.956+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:25.696 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:25.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:25.696 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:25.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:25.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:25.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:25.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:25.403+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:25.697 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:25.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:25.697 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:25.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:25.697 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:25.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:25.697 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:25.396+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:26.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:25.937+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:26.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:25.937+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:26.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:25.937+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:26.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:25.938+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:26.777 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:26.431+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:26.777 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:26.431+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:26.777 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:26.431+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:26.778 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:26.432+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:26.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:26.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:26.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:26.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:26.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:26.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:26.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:26.436+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:26.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.775+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:52:26.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.775+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:52:26.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.775+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f17e801afc0 tx=0x7f17e005d670 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:52:26.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.775+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f17e801afc0 tx=0x7f17e005d670 comp rx=0 tx=0).stop 2024-09-10T03:52:26.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.775+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:52:26.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.775+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:26.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.775+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:26.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.775+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:26.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.775+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80487a0 con 0x7f17d8047ae0 2024-09-10T03:52:26.779 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.775+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:52:26.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.775+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:52:26.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.776+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:52:26.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.776+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:26.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.776+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80485d0 con 0x7f17d8047ae0 2024-09-10T03:52:26.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.777+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f17e8024f60 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:52:26.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.778+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055040 con 0x7f17d8047ae0 2024-09-10T03:52:26.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.778+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8049e80 con 0x7f17d8047ae0 2024-09-10T03:52:26.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:26.779+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064070 con 0x7f17d8047ae0 2024-09-10T03:52:27.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:52:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:52:26.775+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 4200 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:52:27.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:26.896+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:26.896+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:26.896+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:26.897+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:27.420+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:27.420+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:27.420+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:27.421+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:27.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:27.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:27.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:27.410+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:28.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:27.934+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:28.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:27.934+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:28.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:27.934+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:28.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:27.934+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:28.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:28.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:28.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:28.411+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:28.379+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:28.379+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:28.379+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:28.380+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:29.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:28.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:29.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:28.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:29.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:28.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:29.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:28.921+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:29.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:29.397+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:29.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:29.397+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:29.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:29.397+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:29.398+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:29.373+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:29.373+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:29.373+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:29.373+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:30.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:29.913+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:30.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:29.913+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:30.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:29.913+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:29.914+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:30.400+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:30.400+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:30.400+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:30.401+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:30.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:30.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:30.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:30.396+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:30.933+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:30.933+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:30.933+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:30.934+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:31.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.777+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:52:31.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.777+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:52:31.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.777+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f17e8024f60 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:52:31.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.777+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f17e8024f60 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:52:31.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.777+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:52:31.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.777+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:31.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.777+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:31.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.777+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:31.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.777+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80487a0 con 0x7f17d805a5c0 2024-09-10T03:52:31.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.777+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:52:31.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.778+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:52:31.781 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:31.396+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:31.781 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:31.396+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:31.782 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:31.396+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:31.782 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:31.396+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:31.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:31.393+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:31.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:31.393+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:31.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:31.393+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:31.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:31.393+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:31.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.778+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:52:31.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.778+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:31.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.778+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80485d0 con 0x7f17d805a5c0 2024-09-10T03:52:31.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.779+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f17d8018b80 tx=0x7f17e001a790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:52:31.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.780+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8026680 con 0x7f17d805a5c0 2024-09-10T03:52:31.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.780+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8033a90 con 0x7f17d805a5c0 2024-09-10T03:52:31.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.781+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064070 con 0x7f17d805a5c0 2024-09-10T03:52:31.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.839+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:52:31.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:31.839+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:52:32.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:31.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:32.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:31.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:32.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:31.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:31.966+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:32.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:52:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:52:31.777+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 4263 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:52:32.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:32.410+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:32.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:32.410+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:32.410+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:32.410+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:32.412+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:32.412+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:32.412+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:32.412+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:32.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:32.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:32.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:32.949+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:33.361+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:33.361+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:33.361+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:33.362+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:33.377+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:33.377+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:33.377+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:33.378+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:33.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:33.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:33.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:33.974+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:34.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:34.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:34.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:34.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:34.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:34.390+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:34.408+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:34.408+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:34.408+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:34.409+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:35.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:35.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:35.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:35.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:35.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:35.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:35.019+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:35.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:35.430+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:35.430+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:35.430+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:35.431+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:35.383+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:35.383+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:35.383+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:35.384+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:36.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:36.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:36.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:36.009+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:36.779 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:36.473+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:36.780 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:36.473+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:36.780 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:36.473+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:36.780 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:36.474+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:36.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.778+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:52:36.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.778+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:52:36.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.778+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f17d8018b80 tx=0x7f17e001a790 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:52:36.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.778+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f17d8018b80 tx=0x7f17e001a790 comp rx=0 tx=0).stop 2024-09-10T03:52:36.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.778+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:52:36.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.778+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:36.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.778+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:36.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.778+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:36.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.778+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80487a0 con 0x7f17d8047ae0 2024-09-10T03:52:36.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.778+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:52:36.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.778+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:52:36.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:36.335+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:36.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:36.335+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:36.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:36.335+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:36.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:36.336+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:36.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.779+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:52:36.782 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.779+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:36.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.779+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048c70 con 0x7f17d8047ae0 2024-09-10T03:52:36.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.779+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f17e8008550 tx=0x7f17e80421b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:52:36.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.780+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055070 con 0x7f17d8047ae0 2024-09-10T03:52:36.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.780+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b050 con 0x7f17d8047ae0 2024-09-10T03:52:36.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:36.781+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17d8047ae0 2024-09-10T03:52:37.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:36.972+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:37.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:36.972+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:37.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:36.972+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:36.973+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:37.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:52:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:52:36.778+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 4326 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:52:37.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:37.314+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:37.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:37.314+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:37.314+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:37.315+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:37.456+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:37.456+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:37.456+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:37.457+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:38.016+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:38.016+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:38.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:38.016+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:38.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:38.017+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:38.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:38.339+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:38.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:38.339+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:38.339+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:38.340+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:38.469+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:38.469+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:38.469+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:38.470+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:39.059+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:39.059+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:39.059+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:39.060+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:39.380+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:39.380+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:39.380+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:39.381+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:39.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:39.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:39.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:39.488+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:40.072+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:40.072+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:40.072+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:40.073+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:40.496+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:40.496+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:40.496+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:40.496+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:40.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:40.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:40.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:40.419+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:41.431 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:41.121+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:41.431 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:41.121+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:41.431 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:41.121+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:41.431 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:41.122+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:41.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.778+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:52:41.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.778+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:52:41.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.778+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f17e8008550 tx=0x7f17e80421b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:52:41.780 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.778+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f17e8008550 tx=0x7f17e80421b0 comp rx=0 tx=0).stop 2024-09-10T03:52:41.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.778+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:52:41.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.778+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:41.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.778+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:41.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.778+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:41.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.779+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80487a0 con 0x7f17d805a5c0 2024-09-10T03:52:41.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.779+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:52:41.781 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.779+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:52:41.781 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:41.509+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:41.782 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:41.509+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:41.782 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:41.509+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:41.782 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:41.510+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:41.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:41.430+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:41.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:41.430+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:41.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:41.430+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:41.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:41.431+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:41.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.780+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:52:41.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.780+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:41.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.780+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048c70 con 0x7f17d805a5c0 2024-09-10T03:52:41.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.781+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f17e8003fd0 tx=0x7f17e801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:52:41.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.782+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8020e20 con 0x7f17d805a5c0 2024-09-10T03:52:41.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.782+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8034e80 con 0x7f17d805a5c0 2024-09-10T03:52:41.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:41.783+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17d805a5c0 2024-09-10T03:52:42.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:52:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:52:41.780+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 4389 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:52:42.418 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:42.132+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:42.418 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:42.132+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:42.418 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:42.132+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:42.418 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:42.133+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:42.465+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:42.465+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:42.465+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:42.466+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:42.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:42.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:42.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:42.417+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:43.438 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:43.125+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:43.439 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:43.125+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:43.439 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:43.125+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:43.439 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:43.126+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:43.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:43.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:43.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:43.438+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:43.508+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:43.508+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:43.508+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:43.508+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:44.458 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:44.090+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:44.458 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:44.090+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:44.458 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:44.090+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:44.458 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:44.091+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:44.468+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:44.468+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:44.468+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:44.469+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:44.457+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:44.457+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:44.457+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:44.458+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:45.054+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:45.054+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:45.054+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:45.055+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:45.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:45.436+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:45.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:45.436+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:45.436+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:45.436+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:45.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:45.454+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:45.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:45.454+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:45.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:45.454+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:45.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:45.455+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:46.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:46.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:46.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:46.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:46.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:46.075+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:46.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:46.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:46.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:46.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:46.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:46.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:46.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:46.480+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:46.783 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:46.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:46.783 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:46.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:46.783 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:46.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:46.783 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:46.477+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:46.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.780+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:52:46.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.780+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:52:46.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.780+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f17e8003fd0 tx=0x7f17e801c000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:52:46.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.780+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f17e8003fd0 tx=0x7f17e801c000 comp rx=0 tx=0).stop 2024-09-10T03:52:46.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.780+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:52:46.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.780+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:46.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.780+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:46.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.780+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:46.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.780+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80487a0 con 0x7f17d8047ae0 2024-09-10T03:52:46.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.780+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048920 con 0x7f17f0101a40 2024-09-10T03:52:46.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.781+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:52:46.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.781+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:52:46.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.781+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:46.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.781+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80485d0 con 0x7f17d8047ae0 2024-09-10T03:52:46.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.782+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f17e0020000 tx=0x7f17e80144f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:52:46.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.783+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c040 con 0x7f17d8047ae0 2024-09-10T03:52:46.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.783+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0056ec0 con 0x7f17d8047ae0 2024-09-10T03:52:46.785 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:46.784+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d8047ae0 2024-09-10T03:52:47.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:52:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:52:46.781+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 4452 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:52:47.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:47.199+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:52:47.200 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:47.199+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:52:47.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:47.037+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:47.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:47.037+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:47.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:47.037+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:47.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:47.038+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:47.481+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:47.481+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:47.481+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:47.481+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:47.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:47.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:47.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:47.440+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:48.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:47.997+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:48.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:47.997+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:47.997+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:47.998+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:48.458+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:48.458+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:48.458+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:48.459+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:48.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:48.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:48.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:48.497+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:49.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:49.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:49.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:49.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:49.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:49.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:49.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:49.011+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:49.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:49.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:49.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:49.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:49.517+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:49.411+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:49.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:49.411+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:49.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:49.411+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:49.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:49.412+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:50.000+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:50.000+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:50.000+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:50.001+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:50.405+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:50.405+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:50.405+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:50.405+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:50.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:50.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:50.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:50.558+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:50.967+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:50.967+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:50.967+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:50.968+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:51.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.781+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:52:51.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.781+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:52:51.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.781+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f17e0020000 tx=0x7f17e80144f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:52:51.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.781+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f17e0020000 tx=0x7f17e80144f0 comp rx=0 tx=0).stop 2024-09-10T03:52:51.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.781+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:52:51.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.781+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:51.783 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.781+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:51.784 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:51.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:51.784 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:51.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:51.784 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:51.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:51.784 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:51.510+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:51.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.781+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:51.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.781+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80485d0 con 0x7f17d805a5c0 2024-09-10T03:52:51.784 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.781+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048750 con 0x7f17f0101a40 2024-09-10T03:52:51.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:51.387+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:51.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:51.387+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:51.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:51.387+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:51.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:51.387+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:51.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.781+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:52:51.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.782+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:52:51.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.782+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:51.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.782+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80488d0 con 0x7f17d805a5c0 2024-09-10T03:52:51.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.783+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f17e8025010 tx=0x7f17e0009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:52:51.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.783+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8026680 con 0x7f17d805a5c0 2024-09-10T03:52:51.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.784+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b050 con 0x7f17d805a5c0 2024-09-10T03:52:51.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:51.785+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d805a5c0 2024-09-10T03:52:52.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:52:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:52:51.782+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 4515 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:52:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:52.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:52.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:52.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:52.003+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:52.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:52.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:52.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:52.403+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:52.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:52.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:52.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:52.462+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:53.028+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:53.028+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:53.028+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:53.029+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:53.399+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:53.399+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:53.399+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:53.400+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:53.427+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:53.427+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:53.427+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:53.427+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:54.061+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:54.061+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:54.061+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:54.062+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:54.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:54.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:54.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:54.403+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:54.458+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:54.458+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:54.458+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:54.459+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:55.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:55.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:55.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:55.030+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:55.420+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:55.420+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:55.420+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:55.421+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:55.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:55.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:55.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:55.504+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:56.033+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:56.033+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:56.033+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:56.034+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:56.783 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:56.422+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:56.784 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:56.422+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:56.784 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:56.422+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:56.784 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:56.423+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:56.784 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:56.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:56.784 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:56.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:56.784 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:56.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:56.784 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:56.517+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:56.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.782+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:52:56.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.782+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:52:56.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.782+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f17e8025010 tx=0x7f17e0009190 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:52:56.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.782+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f17e8025010 tx=0x7f17e0009190 comp rx=0 tx=0).stop 2024-09-10T03:52:56.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.782+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:52:56.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.782+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:56.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.782+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:56.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.782+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:52:56.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.782+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048710 con 0x7f17d8047ae0 2024-09-10T03:52:56.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.782+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048c70 con 0x7f17f0101a40 2024-09-10T03:52:56.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.783+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:52:56.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.784+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:52:56.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.784+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:52:56.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.784+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048df0 con 0x7f17d8047ae0 2024-09-10T03:52:56.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.785+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f17e8039f10 tx=0x7f17e800cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:52:56.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.785+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055070 con 0x7f17d8047ae0 2024-09-10T03:52:56.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.785+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d8047ae0 2024-09-10T03:52:56.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:52:56.787+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d8047ae0 2024-09-10T03:52:57.069 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:52:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:52:56.783+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 4578 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:52:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:57.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:57.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:57.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:57.068+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:57.729 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:57.440+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:26.377897+0000 front 2024-09-10T03:42:26.377886+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:57.729 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:57.440+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:26.377877+0000 front 2024-09-10T03:42:26.377848+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:57.729 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:57.440+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:26.377830+0000 front 2024-09-10T03:42:26.377915+0000 (oldest deadline 2024-09-10T03:42:52.277759+0000) 2024-09-10T03:52:57.729 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:57.441+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:57.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:57.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:57.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:57.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:57.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:57.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:57.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:57.496+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:58.045+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:58.045+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:58.045+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:58.046+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:58.472+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:58.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:58.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:58.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:58.533+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:52:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:59.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:59.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:59.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:52:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:52:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:52:59.019+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:59.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:52:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:52:59.514+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:52:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:59.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:59.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:59.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:52:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:52:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:52:59.534+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:00.032+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:53:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:00.032+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:53:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:00.032+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:53:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:00.033+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:00.528+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:00.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:00.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:00.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:00.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:00.569+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:01.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:53:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:01.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:53:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:01.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:53:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:01.074+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:01.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:01.559+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:01.785 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:01.526+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:01.785 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:01.526+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:01.785 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:01.526+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:01.786 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:01.526+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:01.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.782+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:53:01.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.782+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:53:01.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.782+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f17e8039f10 tx=0x7f17e800cb30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:53:01.786 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.782+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f17e8039f10 tx=0x7f17e800cb30 comp rx=0 tx=0).stop 2024-09-10T03:53:01.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.782+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:53:01.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.783+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:01.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.783+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:01.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.783+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:01.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.783+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048680 con 0x7f17d805a5c0 2024-09-10T03:53:01.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.783+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048c70 con 0x7f17f0101a40 2024-09-10T03:53:01.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.783+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:53:01.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.784+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:53:01.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.784+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:01.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.785+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048df0 con 0x7f17d805a5c0 2024-09-10T03:53:01.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.786+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f17e0025000 tx=0x7f17e8042170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:53:01.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.786+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028040 con 0x7f17d805a5c0 2024-09-10T03:53:01.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.786+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0056ec0 con 0x7f17d805a5c0 2024-09-10T03:53:01.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:01.787+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d805a5c0 2024-09-10T03:53:02.061 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:53:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:53:01.783+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 4641 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:53:02.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:02.059+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:53:02.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:02.059+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:53:02.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:02.059+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:53:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:02.060+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:02.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:02.558+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:53:02.560 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:02.558+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:53:02.817 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:02.608+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:02.817 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:02.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:02.817 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:02.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:02.817 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:02.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:02.817 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:02.490+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:03.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:03.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:27.461767+0000 front 2024-09-10T03:42:27.461754+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:53:03.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:03.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:27.461798+0000 front 2024-09-10T03:42:27.461789+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:53:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:03.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:27.461777+0000 front 2024-09-10T03:42:27.461726+0000 (oldest deadline 2024-09-10T03:42:52.161669+0000) 2024-09-10T03:53:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:03.043+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:03.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:03.559+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:03.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:03.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:03.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:03.486+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:04.011+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:04.515+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:04.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:04.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:04.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:04.461+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:04.986+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:05.483+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:05.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:05.443+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:05.443+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:05.443+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:05.444+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:06.010+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:06.786 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:06.475+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:06.786 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:06.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:06.786 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:06.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:06.786 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:06.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:06.787 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:06.477+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:06.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.784+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:53:06.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.784+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:53:06.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.784+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f17e0025000 tx=0x7f17e8042170 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:53:06.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.784+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f17e0025000 tx=0x7f17e8042170 comp rx=0 tx=0).stop 2024-09-10T03:53:06.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.784+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:53:06.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.784+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:06.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.784+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:06.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.784+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:06.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.784+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8055670 con 0x7f17d8047ae0 2024-09-10T03:53:06.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.784+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80485d0 con 0x7f17f0101a40 2024-09-10T03:53:06.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.784+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:53:06.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.786+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:53:06.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.786+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:06.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.786+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048c70 con 0x7f17d8047ae0 2024-09-10T03:53:06.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.786+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f17e003c490 tx=0x7f17e0050fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:53:06.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.787+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0015d70 con 0x7f17d8047ae0 2024-09-10T03:53:06.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.787+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e004ab00 con 0x7f17d8047ae0 2024-09-10T03:53:06.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:06.788+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8047ae0 2024-09-10T03:53:07.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:06.972+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:07.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:53:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:53:06.784+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 4704 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:53:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:07.494+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:07.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:07.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:07.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:07.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:07.528+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:08.012+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:08.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:08.485+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:08.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:08.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:08.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:08.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:08.495+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:08.988+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:09.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:09.529+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:09.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:09.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:09.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:09.483+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:09.982+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:10.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:10.522+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:10.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:10.528+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2024-09-10T03:42:29.853978+0000 front 2024-09-10T03:42:29.853931+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:10.528+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2024-09-10T03:42:29.853917+0000 front 2024-09-10T03:42:29.854006+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:10.529+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2024-09-10T03:42:29.853900+0000 front 2024-09-10T03:42:29.853961+0000 (oldest deadline 2024-09-10T03:42:55.153863+0000) 2024-09-10T03:53:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:10.529+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:10.991+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:11.777 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:11.525+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:11.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:11.518+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:11.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.785+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:53:11.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.785+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:53:11.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.786+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f17e003c490 tx=0x7f17e0050fc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:53:11.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.786+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f17e003c490 tx=0x7f17e0050fc0 comp rx=0 tx=0).stop 2024-09-10T03:53:11.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.786+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:53:11.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.786+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:11.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.786+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:11.787 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.786+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:11.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.786+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80485d0 con 0x7f17d805a5c0 2024-09-10T03:53:11.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.786+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048750 con 0x7f17f0101a40 2024-09-10T03:53:11.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.786+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:53:11.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.786+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:53:11.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.786+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:11.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.786+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048c70 con 0x7f17d805a5c0 2024-09-10T03:53:11.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.787+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f17e8000f80 tx=0x7f17e0011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:53:11.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.787+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055070 con 0x7f17d805a5c0 2024-09-10T03:53:11.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.787+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d805a5c0 2024-09-10T03:53:11.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:11.788+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d805a5c0 2024-09-10T03:53:12.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:11.991+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:12.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:53:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:53:11.785+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 4767 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:53:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:12.491+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:12.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:12.474+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:13.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:12.944+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:13.478+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:13.483+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:13.993+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:14.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:14.493+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:14.469+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:14.946+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:15.834 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:15.456+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:15.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:15.443+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:16.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:15.967+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:16.787 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:16.430+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:16.787 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:16.475+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:16.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.786+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:53:16.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.786+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:53:16.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.786+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f17e8000f80 tx=0x7f17e0011e80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:53:16.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.786+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f17e8000f80 tx=0x7f17e0011e80 comp rx=0 tx=0).stop 2024-09-10T03:53:16.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.786+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:53:16.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.786+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:16.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.786+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:16.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.786+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:16.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.786+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048660 con 0x7f17d8047ae0 2024-09-10T03:53:16.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.786+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048c70 con 0x7f17f0101a40 2024-09-10T03:53:16.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.786+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:53:16.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.787+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:53:16.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.787+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:16.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.787+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048df0 con 0x7f17d8047ae0 2024-09-10T03:53:16.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.787+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f17e001afd0 tx=0x7f17e8033230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:53:16.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.787+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c070 con 0x7f17d8047ae0 2024-09-10T03:53:16.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.787+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0014e70 con 0x7f17d8047ae0 2024-09-10T03:53:16.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:16.788+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8047ae0 2024-09-10T03:53:17.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:17.009+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:17.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:53:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:53:16.786+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 4830 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:53:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:17.424+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:17.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:17.520+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:17.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:17.920+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:53:17.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:17.920+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:53:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:17.982+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:18.418+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:18.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:18.556+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:19.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:18.972+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:19.415+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:19.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:19.595+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:20.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:19.972+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:20.585+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:20.454+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:20.454+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:20.454+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:20.455+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:20.999+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:21.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.786+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:53:21.788 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:21.547+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:21.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.786+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:53:21.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.786+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f17e001afd0 tx=0x7f17e8033230 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:53:21.788 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.787+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f17e001afd0 tx=0x7f17e8033230 comp rx=0 tx=0).stop 2024-09-10T03:53:21.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.787+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:53:21.789 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:21.449+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:21.789 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:21.449+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:21.789 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:21.449+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:21.789 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:21.449+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:21.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.787+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:21.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.787+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:21.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.787+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:21.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.787+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048660 con 0x7f17d805a5c0 2024-09-10T03:53:21.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.787+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048c70 con 0x7f17f0101a40 2024-09-10T03:53:21.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.787+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:53:21.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.789+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:53:21.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.789+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:21.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.789+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048df0 con 0x7f17d805a5c0 2024-09-10T03:53:21.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.789+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f17e8001fe0 tx=0x7f17e000b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:53:21.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.790+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e801d5b0 con 0x7f17d805a5c0 2024-09-10T03:53:21.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.790+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803bec0 con 0x7f17d805a5c0 2024-09-10T03:53:21.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:21.791+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d805a5c0 2024-09-10T03:53:22.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:53:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:53:21.787+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 4892 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:53:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:21.953+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:22.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:22.503+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:22.437+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:22.437+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:22.437+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:22.437+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:23.225 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:22.941+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:23.535 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:23.473+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:23.536 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:23.473+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:23.536 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:23.473+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:23.536 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:23.474+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:23.534+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:23.990+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:24.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:24.487+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:24.455+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:24.455+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:24.455+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:24.456+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:25.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:25.006+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:25.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:25.006+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:25.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:25.006+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:25.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:25.007+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:25.492+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:25.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:25.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:25.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:25.408+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:26.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:26.024+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:26.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:26.024+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:26.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:26.024+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:26.024+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:26.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.787+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:53:26.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.787+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:53:26.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.787+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f17e8001fe0 tx=0x7f17e000b930 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:53:26.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.787+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f17e8001fe0 tx=0x7f17e000b930 comp rx=0 tx=0).stop 2024-09-10T03:53:26.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.788+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:53:26.789 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.788+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:26.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.788+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:26.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.788+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:26.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.788+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048660 con 0x7f17d8047ae0 2024-09-10T03:53:26.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.788+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048c70 con 0x7f17f0101a40 2024-09-10T03:53:26.790 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:26.513+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:26.790 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.788+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:53:26.791 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:26.424+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:26.791 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:26.424+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:26.791 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:26.424+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:26.791 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:26.425+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:26.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.788+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:53:26.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.789+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:26.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.789+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048df0 con 0x7f17d8047ae0 2024-09-10T03:53:26.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.789+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f17e001a810 tx=0x7f17e802e680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:53:26.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.791+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c050 con 0x7f17d8047ae0 2024-09-10T03:53:26.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.791+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0022a60 con 0x7f17d8047ae0 2024-09-10T03:53:26.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:26.792+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e003c8c0 con 0x7f17d8047ae0 2024-09-10T03:53:27.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:53:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:53:26.788+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 4954 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:53:27.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:26.986+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:26.986+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:26.986+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:26.987+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:27.544+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:27.406+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:27.406+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:27.406+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:27.407+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:28.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:28.023+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:28.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:28.023+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:28.023+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:28.023+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:28.569+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:28.405+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:28.405+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:28.405+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:28.406+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:29.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:29.026+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:29.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:29.026+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:29.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:29.026+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:29.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:29.027+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:29.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:29.572+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:29.446+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:29.446+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:29.446+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:29.447+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:30.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:29.998+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:30.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:29.998+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:30.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:29.998+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:29.999+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:30.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:30.613+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:30.466+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:30.466+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:30.466+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:30.467+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:31.028+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:31.028+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:31.028+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:31.029+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:31.790 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:31.565+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:31.790 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:31.446+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:31.790 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:31.446+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:31.790 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:31.446+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:31.790 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:31.447+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:31.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.788+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:53:31.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.788+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:53:31.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.788+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f17e001a810 tx=0x7f17e802e680 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:53:31.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.788+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f17e001a810 tx=0x7f17e802e680 comp rx=0 tx=0).stop 2024-09-10T03:53:31.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.788+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:53:31.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.788+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:31.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.789+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:31.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.789+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:31.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.789+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048660 con 0x7f17d805a5c0 2024-09-10T03:53:31.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.789+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048c70 con 0x7f17f0101a40 2024-09-10T03:53:31.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.789+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:53:31.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.790+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:53:31.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.790+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:31.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.790+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048df0 con 0x7f17d805a5c0 2024-09-10T03:53:31.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.791+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f17e00372a0 tx=0x7f17e0011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:53:31.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.793+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c070 con 0x7f17d805a5c0 2024-09-10T03:53:31.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.793+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0022a60 con 0x7f17d805a5c0 2024-09-10T03:53:31.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:31.793+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d805a5c0 2024-09-10T03:53:32.072 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:53:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:53:31.789+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 5016 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:53:32.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:32.071+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:32.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:32.071+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:32.071+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:32.072+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:32.555+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:32.400+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:32.400+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:32.400+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:32.401+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:33.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:33.279+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:53:33.280 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:33.279+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:53:33.418 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:33.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:33.418 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:33.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:33.418 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:33.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:33.419 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:33.109+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:33.583+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:33.417+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:33.417+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:33.417+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:33.417+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:34.065+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:34.065+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:34.065+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:34.065+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:34.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:34.557+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:34.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:34.398+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:34.398+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:34.398+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:34.399+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:35.358 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:35.356+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:35.358 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:35.356+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:35.358 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:35.356+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:35.359 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:35.106+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:35.359 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:35.106+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:35.359 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:35.106+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:35.359 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:35.107+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:35.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:35.357+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:35.564+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:35.564+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:35.564+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:35.565+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:36.061+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:36.061+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:36.061+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:36.062+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:36.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.789+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:53:36.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.789+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:53:36.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.789+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f17e00372a0 tx=0x7f17e0011e80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:53:36.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.789+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f17e00372a0 tx=0x7f17e0011e80 comp rx=0 tx=0).stop 2024-09-10T03:53:36.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.790+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:53:36.791 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.790+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:36.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.790+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:36.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.790+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:36.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.790+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048660 con 0x7f17d8047ae0 2024-09-10T03:53:36.792 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.790+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048c70 con 0x7f17f0101a40 2024-09-10T03:53:36.792 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:36.559+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:36.792 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:36.559+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:36.792 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:36.559+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:36.792 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:36.560+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:36.793 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.790+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:53:36.793 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:36.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:36.793 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:36.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:36.793 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:36.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:36.793 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:36.407+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:36.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.791+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:53:36.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.791+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:36.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.791+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048aa0 con 0x7f17d8047ae0 2024-09-10T03:53:36.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.792+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f17e0015de0 tx=0x7f17e0057ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:53:36.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.794+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c040 con 0x7f17d8047ae0 2024-09-10T03:53:36.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.794+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0022a60 con 0x7f17d8047ae0 2024-09-10T03:53:36.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:36.795+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0037000 con 0x7f17d8047ae0 2024-09-10T03:53:37.049 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:53:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:53:36.790+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 5079 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:53:37.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:37.048+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:37.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:37.048+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:37.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:37.048+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:37.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:37.049+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:37.438+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:37.438+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:37.438+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:37.439+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:37.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:37.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:37.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:37.536+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:38.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:38.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:38.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:38.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:38.019+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:38.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:38.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:38.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:38.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:38.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:38.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:38.548+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:38.415+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:38.415+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:38.415+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:38.416+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:39.057+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:39.057+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:39.057+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:39.058+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:39.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:39.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:39.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:39.524+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:39.458+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:39.458+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:39.458+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:39.459+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:40.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:40.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:40.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:40.068+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:40.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:40.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:40.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:40.479+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:40.502+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:40.502+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:40.502+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:40.503+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:41.437 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:41.099+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:41.438 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:41.099+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:41.438 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:41.099+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:41.438 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:41.100+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:41.792 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:41.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:41.792 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:41.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:41.792 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:41.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:41.793 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:41.480+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:41.793 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:41.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:41.793 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:41.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:41.793 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:41.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:41.793 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:41.436+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:41.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.790+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:53:41.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.790+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:53:41.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.790+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f17e0015de0 tx=0x7f17e0057ba0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:53:41.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.790+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f17e0015de0 tx=0x7f17e0057ba0 comp rx=0 tx=0).stop 2024-09-10T03:53:41.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.790+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8047ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:53:41.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.790+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8047ec0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:41.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.790+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:41.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.790+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:41.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.790+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048660 con 0x7f17d805a5c0 2024-09-10T03:53:41.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.790+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048c70 con 0x7f17f0101a40 2024-09-10T03:53:41.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.791+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:53:41.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.791+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:53:41.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.792+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:41.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.792+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048df0 con 0x7f17d805a5c0 2024-09-10T03:53:41.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.792+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f17e001e950 tx=0x7f17e0018690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:53:41.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.792+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c1e0 con 0x7f17d805a5c0 2024-09-10T03:53:41.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.792+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e003ece0 con 0x7f17d805a5c0 2024-09-10T03:53:41.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:41.793+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d805a5c0 2024-09-10T03:53:42.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:53:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:53:41.790+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 5142 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:53:42.458 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:42.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:42.458 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:42.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:42.458 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:42.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:42.459 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:42.096+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:42.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:42.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:42.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:42.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:42.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:42.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:42.456+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:42.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:42.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:42.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:42.468+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:43.445 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:43.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:43.446 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:43.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:43.446 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:43.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:43.446 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:43.118+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:43.443+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:43.443+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:43.443+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:43.444+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:43.466+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:43.466+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:43.466+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:43.466+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:44.456 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:44.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:44.456 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:44.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:44.456 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:44.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:44.456 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:44.079+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:44.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:44.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:44.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:44.468+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:44.455+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:44.455+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:44.455+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:44.455+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:45.423 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:45.104+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:45.423 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:45.104+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:45.423 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:45.104+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:45.423 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:45.105+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:45.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:45.448+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:45.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:45.448+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:45.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:45.448+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:45.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:45.449+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:45.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:45.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:45.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:45.422+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:46.397 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:46.138+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:46.397 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:46.138+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:46.397 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:46.138+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:46.397 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:46.138+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:46.794 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:46.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:46.794 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:46.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:46.794 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:46.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:46.794 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:46.410+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:46.794 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:46.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:46.794 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:46.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:46.795 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:46.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:46.795 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:46.396+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:46.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.792+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:53:46.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.793+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:53:46.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.793+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f17e001e950 tx=0x7f17e0018690 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:53:46.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.793+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f17e001e950 tx=0x7f17e0018690 comp rx=0 tx=0).stop 2024-09-10T03:53:46.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.793+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:53:46.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.793+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:46.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.793+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:46.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.793+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:46.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.793+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048660 con 0x7f17d8047ae0 2024-09-10T03:53:46.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.793+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047f60 con 0x7f17f0101a40 2024-09-10T03:53:46.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.793+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:53:46.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.793+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:53:46.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.793+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:46.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.794+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80480e0 con 0x7f17d8047ae0 2024-09-10T03:53:46.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.794+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f17d800a1a0 tx=0x7f17e001a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:53:46.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.794+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055070 con 0x7f17d8047ae0 2024-09-10T03:53:46.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.794+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b020 con 0x7f17d8047ae0 2024-09-10T03:53:46.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:46.796+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17d8047ae0 2024-09-10T03:53:47.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:53:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:53:46.792+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 5205 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:53:47.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:47.406+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:47.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:47.406+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:47.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:47.406+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:47.407+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:47.185+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:47.185+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:47.185+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:47.186+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:47.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:47.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:47.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:47.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:47.367+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:48.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:48.408+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:48.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:48.408+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:48.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:48.408+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:48.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:48.409+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:48.179+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:48.179+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:48.179+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:48.179+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:48.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:48.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:48.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:48.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:48.358+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:48.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:48.639+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:53:48.640 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:48.639+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:53:49.442 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:49.180+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:49.443 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:49.180+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:49.443 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:49.180+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:49.443 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:49.180+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:49.443 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:49.397+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:49.443 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:49.397+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:49.443 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:49.397+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:49.443 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:49.398+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:49.441+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:49.441+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:49.441+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:49.442+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:50.471 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:50.359+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:50.471 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:50.359+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:50.471 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:50.359+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:50.472 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:50.359+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:50.472 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:50.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:50.472 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:50.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:50.472 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:50.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:50.472 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:50.150+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:50.470+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:50.470+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:50.470+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:50.471+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:51.465 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:51.323+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:51.465 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:51.323+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:51.465 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:51.323+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:51.465 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:51.323+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:51.466 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:51.129+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:51.466 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:51.129+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:51.466 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:51.129+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:51.466 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:51.130+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:51.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.793+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:53:51.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.793+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:53:51.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.793+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f17d800a1a0 tx=0x7f17e001a8c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:53:51.794 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.793+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f17d800a1a0 tx=0x7f17e001a8c0 comp rx=0 tx=0).stop 2024-09-10T03:53:51.794 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:51.464+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:51.795 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:51.464+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:51.795 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:51.464+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:51.795 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:51.464+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:51.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.793+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:53:51.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.793+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:51.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.793+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:51.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.793+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:51.795 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.793+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048660 con 0x7f17d805a5c0 2024-09-10T03:53:51.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.793+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047f60 con 0x7f17f0101a40 2024-09-10T03:53:51.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.793+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:53:51.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.795+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:53:51.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.795+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:51.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.795+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80480e0 con 0x7f17d805a5c0 2024-09-10T03:53:51.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.795+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f17d8017470 tx=0x7f17e8042190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:53:51.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.795+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e800d980 con 0x7f17d805a5c0 2024-09-10T03:53:51.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.795+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e801d820 con 0x7f17d805a5c0 2024-09-10T03:53:51.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:51.796+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c050 con 0x7f17d805a5c0 2024-09-10T03:53:52.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:53:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:53:51.793+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 5268 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:53:52.511 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:52.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:52.512 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:52.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:52.512 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:52.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:52.512 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:52.330+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:52.512 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:52.115+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:52.512 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:52.115+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:52.512 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:52.115+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:52.512 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:52.116+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:52.510+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:52.510+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:52.510+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:52.511+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:53.537 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:53.290+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:53.537 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:53.290+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:53.537 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:53.290+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:53.538 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:53.290+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:53.538 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:53.099+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:53.538 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:53.099+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:53.538 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:53.099+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:53.538 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:53.100+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:53.536+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:53.536+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:53.536+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:53.537+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:54.513 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:54.294+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:54.513 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:54.294+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:54.513 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:54.294+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:54.513 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:54.295+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:54.514 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:54.136+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:54.514 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:54.136+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:54.514 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:54.136+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:54.514 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:54.136+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:54.784 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:54.512+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:54.784 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:54.512+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:54.784 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:54.512+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:54.784 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:54.512+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:55.467 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:55.092+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:55.467 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:55.092+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:55.467 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:55.092+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:55.467 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:55.093+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:55.468 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:55.324+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:55.468 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:55.324+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:55.468 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:55.324+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:55.468 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:55.324+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:55.465+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:55.465+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:55.465+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:55.466+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:56.311 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:56.058+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:56.311 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:56.058+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:56.311 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:56.058+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:56.311 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:56.059+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:56.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:56.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:56.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:56.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:56.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:56.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:56.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:56.494+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:56.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:56.310+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:56.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:56.310+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:56.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:56.310+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:56.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:56.310+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:56.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.793+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:53:56.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.793+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:53:56.796 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.794+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f17d8017470 tx=0x7f17e8042190 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:53:56.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.794+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f17d8017470 tx=0x7f17e8042190 comp rx=0 tx=0).stop 2024-09-10T03:53:56.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.794+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:53:56.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.794+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:56.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.794+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:56.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.794+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:53:56.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.794+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047ff0 con 0x7f17d8047ae0 2024-09-10T03:53:56.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.794+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17f0101a40 2024-09-10T03:53:56.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.794+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:53:56.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.795+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:53:56.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.795+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:53:56.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.795+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80488d0 con 0x7f17d8047ae0 2024-09-10T03:53:56.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.796+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f17e802af10 tx=0x7f17e800f050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:53:56.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.797+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e804e9d0 con 0x7f17d8047ae0 2024-09-10T03:53:56.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.797+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8001d30 con 0x7f17d8047ae0 2024-09-10T03:53:56.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:53:56.798+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c050 con 0x7f17d8047ae0 2024-09-10T03:53:57.053 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:53:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:53:56.794+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 5331 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:53:57.312 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:57.051+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:57.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:57.051+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:57.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:57.051+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:57.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:57.052+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:57.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:57.461+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:57.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:57.461+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:57.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:57.461+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:57.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:57.461+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:57.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:57.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:57.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:57.311+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:58.009+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:58.009+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:58.009+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:58.009+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:58.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:58.337+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:58.337+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:58.337+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:58.338+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:53:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:58.461+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:58.461+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:58.461+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:58.462+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:59.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:58.986+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:59.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:58.986+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:59.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:58.986+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:53:59.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:58.987+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:59.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:59.481+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:59.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:59.481+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:59.481+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:53:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:53:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:53:59.481+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:53:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:59.363+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:59.363+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:59.363+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:53:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:53:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:53:59.363+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:59.938+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:59.938+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:59.938+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:53:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:53:59.939+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:00.463+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:00.463+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:00.463+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:00.463+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:00.409+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:00.409+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:00.409+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:00.409+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:00.946+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:00.946+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:00.946+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:00.947+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:01.796 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:01.451+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:01.796 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:01.451+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:01.797 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:01.451+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:01.797 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:01.452+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:01.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.795+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:54:01.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.795+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:54:01.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.795+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f17e802af10 tx=0x7f17e800f050 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:54:01.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.795+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f17e802af10 tx=0x7f17e800f050 comp rx=0 tx=0).stop 2024-09-10T03:54:01.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.795+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:54:01.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.795+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:01.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.795+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:01.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.795+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:01.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.795+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047ff0 con 0x7f17d805a5c0 2024-09-10T03:54:01.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.795+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17f0101a40 2024-09-10T03:54:01.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.795+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:54:01.798 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:01.459+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:01.799 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:01.459+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:01.799 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:01.459+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:01.799 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:01.460+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:01.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.796+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:54:01.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.796+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:01.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.796+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80488d0 con 0x7f17d805a5c0 2024-09-10T03:54:01.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.796+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f17e0006d80 tx=0x7f17e8004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:54:01.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.797+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0057e00 con 0x7f17d805a5c0 2024-09-10T03:54:01.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.798+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e003be50 con 0x7f17d805a5c0 2024-09-10T03:54:01.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:01.799+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d805a5c0 2024-09-10T03:54:02.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:54:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:54:01.795+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 5394 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:54:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:01.980+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:01.980+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:01.980+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:01.980+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:02.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:02.439+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:02.439+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:02.439+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:02.439+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:02.478+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:02.478+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:02.478+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:02.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:02.479+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:03.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:03.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:03.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:03.002+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:03.472+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:03.472+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:03.472+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:03.473+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:03.472+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:03.472+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:03.472+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:03.473+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:04.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:03.999+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:54:04.000 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:03.999+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:54:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:03.953+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:03.953+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:03.953+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:03.953+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:04.501+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:04.501+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:04.501+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:04.502+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:04.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:04.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:04.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:04.493+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:04.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:04.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:04.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:04.904+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:05.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:05.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:05.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:05.479+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:05.492+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:05.492+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:05.492+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:05.493+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:05.872+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:05.872+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:05.872+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:05.873+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:06.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.796+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:54:06.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.796+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:54:06.797 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.796+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f17e0006d80 tx=0x7f17e8004840 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:54:06.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.796+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f17e0006d80 tx=0x7f17e8004840 comp rx=0 tx=0).stop 2024-09-10T03:54:06.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.796+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:54:06.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.796+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:06.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.796+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:06.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.796+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:06.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.796+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047ff0 con 0x7f17d8047ae0 2024-09-10T03:54:06.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.796+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17f0101a40 2024-09-10T03:54:06.798 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:06.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:06.798 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:06.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:06.798 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:06.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:06.799 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:06.477+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:06.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.797+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:54:06.799 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:06.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:06.799 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:06.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:06.799 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:06.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:06.799 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:06.440+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:06.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.797+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:54:06.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.797+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:06.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.797+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048aa0 con 0x7f17d8047ae0 2024-09-10T03:54:06.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.798+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f17e0019040 tx=0x7f17e003fd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:54:06.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.799+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028040 con 0x7f17d8047ae0 2024-09-10T03:54:06.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.799+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e003be50 con 0x7f17d8047ae0 2024-09-10T03:54:06.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:06.800+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d8047ae0 2024-09-10T03:54:07.073 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:54:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:54:06.796+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 5457 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:54:07.073 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:06.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:07.073 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:06.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:07.073 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:06.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:07.073 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:06.853+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:07.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:07.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:07.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:07.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:07.477+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:07.473+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:07.473+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:07.473+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:07.473+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:07.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:07.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:07.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:08.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:07.871+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:08.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:08.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:08.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:08.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:08.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:08.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:08.484+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:08.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:08.431+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:08.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:08.431+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:08.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:08.431+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:08.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:08.432+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:09.153 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:08.843+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:09.153 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:08.843+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:09.153 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:08.843+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:09.153 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:08.844+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:09.497 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:09.396+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:09.497 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:09.396+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:09.497 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:09.396+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:09.497 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:09.396+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:09.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:09.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:09.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:09.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:09.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:09.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:09.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:09.496+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:09.849+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:09.849+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:09.849+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:09.849+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:10.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:10.377+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:10.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:10.377+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:10.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:10.377+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:10.378+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:10.447+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:10.447+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:10.447+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:10.448+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:10.839+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:10.839+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:10.839+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:10.840+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:11.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.797+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:54:11.798 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.797+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:54:11.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.797+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f17e0019040 tx=0x7f17e003fd50 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:54:11.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.797+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f17e0019040 tx=0x7f17e003fd50 comp rx=0 tx=0).stop 2024-09-10T03:54:11.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.797+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:54:11.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.797+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:11.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.797+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:11.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.797+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:11.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.797+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80487e0 con 0x7f17d805a5c0 2024-09-10T03:54:11.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.797+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047ff0 con 0x7f17f0101a40 2024-09-10T03:54:11.800 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:11.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:11.800 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:11.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:11.800 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:11.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:11.800 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:11.496+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:11.800 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:11.395+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:11.800 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:11.395+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:11.800 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:11.396+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:11.800 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:11.396+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:11.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.797+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:54:11.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.798+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:54:11.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.798+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:11.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.798+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048aa0 con 0x7f17d805a5c0 2024-09-10T03:54:11.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.798+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f17d80101f0 tx=0x7f17e0035760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:54:11.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.799+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e803f980 con 0x7f17d805a5c0 2024-09-10T03:54:11.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.800+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8001d30 con 0x7f17d805a5c0 2024-09-10T03:54:11.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:11.800+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c050 con 0x7f17d805a5c0 2024-09-10T03:54:12.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:54:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:54:11.797+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 5520 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:54:12.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:11.834+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:11.834+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:11.834+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:11.835+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:12.376+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:12.376+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:12.376+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:12.377+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:12.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:12.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:12.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:12.487+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:13.217 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:12.858+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:13.217 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:12.858+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:13.217 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:12.858+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:13.217 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:12.858+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:13.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:13.420+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:13.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:13.420+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:13.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:13.420+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:13.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:13.420+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:13.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:13.452+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:13.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:13.452+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:13.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:13.452+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:13.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:13.453+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:14.262 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:13.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:14.262 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:13.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:14.262 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:13.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:14.262 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:13.869+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:14.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:14.427+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:14.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:14.427+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:14.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:14.427+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:14.427+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:14.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:14.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:14.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:14.503+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:14.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:14.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:14.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:14.870+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:15.487+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:15.487+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:15.487+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:15.488+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:15.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:15.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:15.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:15.468+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:16.288 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:15.910+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:16.289 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:15.910+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:16.289 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:15.910+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:16.289 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:15.911+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:16.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:16.477+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:16.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:16.477+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:16.477+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:16.478+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:16.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:16.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:16.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:16.509+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:16.799 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.797+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:54:16.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.797+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:54:16.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.797+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f17d80101f0 tx=0x7f17e0035760 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:54:16.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.797+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f17d80101f0 tx=0x7f17e0035760 comp rx=0 tx=0).stop 2024-09-10T03:54:16.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.797+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:54:16.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.797+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:16.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.797+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:16.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.797+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:16.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.797+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80487e0 con 0x7f17d8047ae0 2024-09-10T03:54:16.800 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.797+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047ff0 con 0x7f17f0101a40 2024-09-10T03:54:16.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.798+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:54:16.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.798+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:54:16.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.798+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:16.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.798+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048170 con 0x7f17d8047ae0 2024-09-10T03:54:16.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.799+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f17e0019c30 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:54:16.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.799+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028070 con 0x7f17d8047ae0 2024-09-10T03:54:16.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.799+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0065e50 con 0x7f17d8047ae0 2024-09-10T03:54:16.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:16.800+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8047ae0 2024-09-10T03:54:17.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:54:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:54:16.797+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 5583 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:54:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:16.910+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:16.910+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:16.910+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:16.911+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:17.492+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:17.492+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:17.492+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:17.492+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:17.529+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:17.529+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:17.529+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:17.530+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:17.909+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:17.909+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:18.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:17.909+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:18.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:17.910+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:18.848 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:18.537+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:18.848 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:18.537+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:18.849 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:18.537+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:18.849 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:18.538+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:18.849 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:18.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:18.849 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:18.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:18.849 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:18.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:18.849 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:18.561+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:19.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:18.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:19.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:18.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:19.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:18.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:19.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:18.898+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:19.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:19.358+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:54:19.360 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:19.358+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:54:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:19.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:19.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:19.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:19.574+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:19.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:19.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:19.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:19.591+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:20.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:19.938+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:19.938+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:19.938+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:19.939+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:20.536+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:20.536+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:20.536+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:20.536+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:20.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:20.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:20.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:20.633+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:20.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:20.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:20.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:20.990+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:21.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.798+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:54:21.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.798+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:54:21.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.798+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f17e0019c30 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:54:21.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.798+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f17e0019c30 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:54:21.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.798+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:54:21.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.798+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:21.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.798+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:21.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.798+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:21.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.798+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047ff0 con 0x7f17d805a5c0 2024-09-10T03:54:21.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.798+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048730 con 0x7f17f0101a40 2024-09-10T03:54:21.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.798+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:54:21.802 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:21.504+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:21.802 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:21.504+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:21.802 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:21.504+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:21.802 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:21.505+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:21.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:21.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:21.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:21.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:21.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:21.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:21.803 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:21.632+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:21.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.799+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:54:21.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.799+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:21.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.799+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80488b0 con 0x7f17d805a5c0 2024-09-10T03:54:21.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.799+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f17e0006d80 tx=0x7f17e0053710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:54:21.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.801+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c040 con 0x7f17d805a5c0 2024-09-10T03:54:21.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.801+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0016ec0 con 0x7f17d805a5c0 2024-09-10T03:54:21.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:21.802+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d805a5c0 2024-09-10T03:54:22.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:54:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:54:21.799+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 5646 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:54:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:22.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:22.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:22.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:22.011+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:22.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:22.498+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:22.498+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:22.498+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:22.498+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:22.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:22.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:22.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:22.634+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:23.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:22.993+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:23.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:22.993+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:23.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:22.993+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:23.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:22.994+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:23.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:23.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:23.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:23.625+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:23.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:23.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:23.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:23.532+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:23.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:23.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:23.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:23.976+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:24.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:24.662+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:24.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:24.662+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:24.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:24.662+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:24.663+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:24.561+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:24.561+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:24.561+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:24.562+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:25.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:25.009+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:25.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:25.009+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:25.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:25.009+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:25.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:25.010+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:25.575+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:25.575+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:25.575+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:25.576+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:25.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:25.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:25.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:25.711+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:26.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:26.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:26.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:26.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:26.018+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:26.801 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.800+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:54:26.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.800+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:54:26.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.800+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f17e0006d80 tx=0x7f17e0053710 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:54:26.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.800+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f17e0006d80 tx=0x7f17e0053710 comp rx=0 tx=0).stop 2024-09-10T03:54:26.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.800+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:54:26.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.800+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:26.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.800+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:26.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.800+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:26.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.800+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80486a0 con 0x7f17d8047ae0 2024-09-10T03:54:26.802 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.800+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048820 con 0x7f17f0101a40 2024-09-10T03:54:26.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.801+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:54:26.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.801+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:54:26.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.801+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:26.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.801+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048170 con 0x7f17d8047ae0 2024-09-10T03:54:26.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.802+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f17e0018700 tx=0x7f17e0041540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:54:26.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.803+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028070 con 0x7f17d8047ae0 2024-09-10T03:54:26.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.803+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0065e50 con 0x7f17d8047ae0 2024-09-10T03:54:26.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:26.804+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8047ae0 2024-09-10T03:54:27.023 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:54:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:54:26.801+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 5709 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:54:27.023 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:26.584+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:27.023 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:26.584+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:27.023 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:26.584+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:27.023 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:26.585+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:27.024 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:26.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:27.024 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:26.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:27.024 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:26.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:27.024 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:26.710+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:27.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:27.021+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:27.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:27.021+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:27.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:27.021+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:27.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:27.022+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:27.676+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:27.676+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:27.676+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:27.677+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:27.557+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:27.557+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:27.557+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:27.558+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:28.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:27.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:28.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:27.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:28.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:27.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:27.985+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:29.006 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:28.725+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:29.006 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:28.725+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:29.006 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:28.725+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:29.006 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:28.725+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:29.006 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:28.601+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:29.006 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:28.601+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:29.006 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:28.601+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:29.006 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:28.602+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:29.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:29.005+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:29.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:29.005+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:29.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:29.005+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:29.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:29.006+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:29.977 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:29.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:29.977 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:29.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:29.977 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:29.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:29.977 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:29.755+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:29.978 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:29.599+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:29.978 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:29.599+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:29.978 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:29.599+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:29.978 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:29.600+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:30.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:29.976+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:29.976+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:29.976+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:29.977+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:31.018 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:30.629+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:31.018 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:30.629+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:31.018 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:30.629+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:31.019 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:30.629+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:31.019 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:30.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:31.019 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:30.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:31.019 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:30.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:31.019 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:30.774+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:31.016+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:31.016+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:31.016+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:31.017+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:31.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.801+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:54:31.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.801+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:54:31.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.801+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f17e0018700 tx=0x7f17e0041540 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:54:31.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.801+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f17e0018700 tx=0x7f17e0041540 comp rx=0 tx=0).stop 2024-09-10T03:54:31.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.801+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:54:31.803 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.801+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:31.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.801+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:31.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.801+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:31.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.801+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80486a0 con 0x7f17d805a5c0 2024-09-10T03:54:31.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.801+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048820 con 0x7f17f0101a40 2024-09-10T03:54:31.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.801+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:54:31.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.802+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:54:31.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.802+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:31.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.802+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80491b0 con 0x7f17d805a5c0 2024-09-10T03:54:31.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.802+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f17e800c4f0 tx=0x7f17e001ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:54:31.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.803+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055040 con 0x7f17d805a5c0 2024-09-10T03:54:31.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.803+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d805a5c0 2024-09-10T03:54:31.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:31.804+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c050 con 0x7f17d805a5c0 2024-09-10T03:54:32.005 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:31.603+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:32.005 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:31.603+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:32.005 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:31.603+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:32.005 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:31.603+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:32.006 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:31.813+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:32.006 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:31.813+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:32.006 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:31.813+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:32.006 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:31.814+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:32.006 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:54:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:54:31.802+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 5772 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:54:32.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:32.004+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:32.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:32.004+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:32.004+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:32.004+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:32.819+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:32.820+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:32.820+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:32.820+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:32.572+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:32.572+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:32.572+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:32.573+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:33.024+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:33.024+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:33.024+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:33.025+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:33.799 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:33.525+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:33.799 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:33.525+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:33.799 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:33.525+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:33.799 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:33.525+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:34.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:33.998+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:34.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:33.998+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:34.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:33.998+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:33.999+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:33.797+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:33.797+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:33.797+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:33.798+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:34.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:34.720+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:54:34.721 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:34.720+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:54:34.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:34.765+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:34.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:34.765+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:34.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:34.765+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:34.765+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:34.519+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:34.519+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:34.519+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:34.520+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:35.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:35.030+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:35.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:35.030+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:35.030+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:35.031+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:35.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:35.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:35.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:35.749+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:35.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:35.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:35.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:35.566+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:36.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:36.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:36.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:36.076+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:36.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.802+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:54:36.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.802+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:54:36.804 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.802+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f17e800c4f0 tx=0x7f17e001ef40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:54:36.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.802+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f17e800c4f0 tx=0x7f17e001ef40 comp rx=0 tx=0).stop 2024-09-10T03:54:36.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.802+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:54:36.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.802+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:36.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.802+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:36.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.802+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:36.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.802+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80486a0 con 0x7f17d8047ae0 2024-09-10T03:54:36.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.802+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048820 con 0x7f17f0101a40 2024-09-10T03:54:36.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.803+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:54:36.805 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:36.756+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:36.805 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:36.756+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:36.806 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:36.756+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:36.806 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:36.756+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:36.806 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:36.552+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:36.806 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:36.552+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:36.806 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:36.552+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:36.806 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:36.553+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:36.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.803+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:54:36.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.803+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:36.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.803+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80491b0 con 0x7f17d8047ae0 2024-09-10T03:54:36.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.804+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f17e003f250 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:54:36.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.805+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028040 con 0x7f17d8047ae0 2024-09-10T03:54:36.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.805+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0016d10 con 0x7f17d8047ae0 2024-09-10T03:54:36.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:36.805+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d8047ae0 2024-09-10T03:54:37.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:54:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:54:36.802+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 5835 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:54:37.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:37.090+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:37.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:37.090+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:37.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:37.090+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:37.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:37.091+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:38.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:37.747+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:38.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:37.747+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:38.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:37.747+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:38.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:37.747+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:37.602+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:37.602+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:37.602+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:37.602+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:38.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:38.132+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:38.132+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:38.132+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:38.133+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:39.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:38.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:39.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:38.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:39.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:38.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:38.745+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:38.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:38.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:38.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:38.587+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:39.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:39.098+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:39.098+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:39.098+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:39.098+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:40.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:39.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:40.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:39.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:39.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:39.700+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:39.583+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:39.583+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:39.583+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:39.583+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:40.557 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:40.104+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:40.557 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:40.104+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:40.557 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:40.104+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:40.557 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:40.105+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:40.555+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:40.555+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:40.555+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:40.556+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:40.674+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:40.674+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:40.674+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:40.674+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:41.557 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:41.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:41.558 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:41.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:41.558 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:41.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:41.558 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:41.081+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:41.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.803+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:54:41.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.803+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:54:41.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.803+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f17e003f250 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:54:41.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.803+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f17e003f250 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:54:41.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.803+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:54:41.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.803+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:41.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.803+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:41.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.803+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:41.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.803+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80486a0 con 0x7f17d805a5c0 2024-09-10T03:54:41.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.803+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048820 con 0x7f17f0101a40 2024-09-10T03:54:41.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.803+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:54:41.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.804+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:54:41.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.804+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:41.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.804+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048170 con 0x7f17d805a5c0 2024-09-10T03:54:41.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.804+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f17e00145c0 tx=0x7f17e002f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:54:41.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.805+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c040 con 0x7f17d805a5c0 2024-09-10T03:54:41.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.805+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0018db0 con 0x7f17d805a5c0 2024-09-10T03:54:41.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:41.806+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d805a5c0 2024-09-10T03:54:41.808 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:41.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:41.808 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:41.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:41.808 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:41.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:41.808 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:41.704+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:41.808 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:41.556+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:41.809 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:41.556+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:41.809 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:41.556+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:41.809 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:41.557+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:41.809 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:54:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:54:41.803+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 5898 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:54:42.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:42.120+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:42.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:42.120+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:42.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:42.120+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:42.121+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:43.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:42.585+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:42.585+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:42.585+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:42.586+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:42.721+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:42.721+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:42.721+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:42.722+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:43.561 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:43.119+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:43.561 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:43.119+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:43.561 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:43.119+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:43.561 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:43.120+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:43.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:43.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:43.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:43.606+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:43.768+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:43.768+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:43.768+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:43.769+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:44.122+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:44.122+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:44.122+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:44.122+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:44.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:44.724+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:44.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:44.724+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:44.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:44.724+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:44.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:44.725+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:44.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:44.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:44.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:44.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:44.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:44.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:44.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:44.571+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:45.081+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:45.081+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:45.081+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:45.082+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:45.735 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:45.553+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:45.735 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:45.553+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:45.735 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:45.553+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:45.735 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:45.554+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:46.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:45.733+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:46.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:45.733+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:46.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:45.733+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:45.734+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:46.545 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:46.082+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:46.546 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:46.082+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:46.546 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:46.082+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:46.546 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:46.083+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:46.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.804+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:54:46.805 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.804+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:54:46.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.804+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f17e00145c0 tx=0x7f17e002f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:54:46.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.804+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f17e00145c0 tx=0x7f17e002f040 comp rx=0 tx=0).stop 2024-09-10T03:54:46.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.804+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:54:46.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.804+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:46.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.804+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:46.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.804+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:46.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.804+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047ff0 con 0x7f17d8047ae0 2024-09-10T03:54:46.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.804+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80486a0 con 0x7f17f0101a40 2024-09-10T03:54:46.806 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:46.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:46.807 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:46.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:46.807 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:46.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:46.807 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:46.744+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:46.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.804+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:54:46.807 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:46.544+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:46.807 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:46.544+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:46.807 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:46.544+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:46.808 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:46.545+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:46.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.804+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:54:46.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.805+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:46.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.805+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048820 con 0x7f17d8047ae0 2024-09-10T03:54:46.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.805+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f17e802ce90 tx=0x7f17e0011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:54:46.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.805+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e803f960 con 0x7f17d8047ae0 2024-09-10T03:54:46.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.806+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b080 con 0x7f17d8047ae0 2024-09-10T03:54:46.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:46.806+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c050 con 0x7f17d8047ae0 2024-09-10T03:54:47.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:54:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:54:46.804+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 5961 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:54:47.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:47.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:47.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:47.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:47.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:47.043+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:47.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:47.496+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:47.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:47.496+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:47.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:47.496+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:47.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:47.497+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:48.075 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:47.783+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:48.075 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:47.783+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:48.075 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:47.783+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:48.075 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:47.784+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:48.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:48.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:48.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:48.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:48.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:48.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:48.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:48.074+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:48.768 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:48.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:48.768 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:48.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:48.769 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:48.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:48.769 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:48.480+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:49.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:48.767+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:49.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:48.767+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:49.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:48.767+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:48.768+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:49.483 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:49.114+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:49.483 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:49.114+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:49.484 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:49.114+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:49.484 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:49.115+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:49.734 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:49.482+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:49.734 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:49.482+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:49.734 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:49.482+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:49.735 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:49.482+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:50.000 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:49.733+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:50.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:49.733+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:50.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:49.733+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:50.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:49.733+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:50.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:50.208+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:54:50.209 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:50.208+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:54:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:50.161+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:50.161+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:50.161+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:50.162+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:50.488+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:50.488+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:50.488+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:50.488+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:50.725+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:50.726+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:50.726+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:50.726+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:51.529 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:51.157+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:51.529 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:51.157+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:51.529 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:51.157+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:51.529 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:51.158+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:51.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.805+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:54:51.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.805+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:54:51.806 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.805+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f17e802ce90 tx=0x7f17e0011ca0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:54:51.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.805+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f17e802ce90 tx=0x7f17e0011ca0 comp rx=0 tx=0).stop 2024-09-10T03:54:51.807 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:51.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:51.807 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:51.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:51.807 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:51.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:51.807 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:51.679+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:51.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.805+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:54:51.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.805+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:51.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.805+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:51.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.805+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:51.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.805+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047f60 con 0x7f17d805a5c0 2024-09-10T03:54:51.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.805+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80486a0 con 0x7f17f0101a40 2024-09-10T03:54:51.808 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:51.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:51.808 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:51.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:51.808 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:51.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:51.808 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:51.528+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:51.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.806+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:54:51.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.806+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:54:51.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.806+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:51.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.806+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048aa0 con 0x7f17d805a5c0 2024-09-10T03:54:51.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.807+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f17d80170a0 tx=0x7f17e803f660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:54:51.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.808+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055030 con 0x7f17d805a5c0 2024-09-10T03:54:51.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.808+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d805a5c0 2024-09-10T03:54:51.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:51.809+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c040 con 0x7f17d805a5c0 2024-09-10T03:54:52.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:54:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:54:51.805+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6024 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:54:52.573 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:52.165+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:52.573 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:52.165+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:52.573 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:52.165+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:52.573 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:52.166+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:52.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:52.694+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:52.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:52.694+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:52.694+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:52.695+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:52.571+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:52.571+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:52.571+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:52.572+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:53.546 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:53.172+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:53.547 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:53.172+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:53.547 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:53.172+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:53.547 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:53.173+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:53.544+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:53.544+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:53.544+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:53.545+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:53.715+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:53.715+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:53.715+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:53.715+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:54.550 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:54.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:54.551 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:54.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:54.551 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:54.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:54.551 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:54.150+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:54.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:54.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:54.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:54.549+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:54.740+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:54.740+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:54.740+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:54.741+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:55.564 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:55.196+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:55.564 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:55.196+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:55.564 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:55.196+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:55.564 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:55.197+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:55.561+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:55.561+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:55.561+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:55.562+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:55.767+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:55.767+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:55.767+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:55.768+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:56.573 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:56.215+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:56.573 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:56.215+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:56.573 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:56.215+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:56.573 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:56.216+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:56.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.804+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:54:56.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.804+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:54:56.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.804+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f17d80170a0 tx=0x7f17e803f660 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:54:56.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.804+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f17d80170a0 tx=0x7f17e803f660 comp rx=0 tx=0).stop 2024-09-10T03:54:56.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.805+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:54:56.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.805+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:56.807 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.805+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:56.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.805+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:54:56.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.805+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047f60 con 0x7f17d8047ae0 2024-09-10T03:54:56.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.805+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80486a0 con 0x7f17f0101a40 2024-09-10T03:54:56.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.805+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:54:56.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.806+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:54:56.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.806+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:54:56.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.806+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048820 con 0x7f17d8047ae0 2024-09-10T03:54:56.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.806+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f17e8064000 tx=0x7f17e800f050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:54:56.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.807+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8026680 con 0x7f17d8047ae0 2024-09-10T03:54:56.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.807+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e805be70 con 0x7f17d8047ae0 2024-09-10T03:54:56.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:54:56.808+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c040 con 0x7f17d8047ae0 2024-09-10T03:54:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:56.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:56.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:56.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:56.571+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:56.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:56.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:56.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:56.788+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:56.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:54:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:54:56.805+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6087 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:54:57.537 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:57.215+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:57.537 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:57.215+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:57.537 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:57.215+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:57.537 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:57.216+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:57.806 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:57.532+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:57.806 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:57.532+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:57.807 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:57.532+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:57.807 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:57.533+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:58.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:57.804+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:58.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:57.804+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:58.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:57.804+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:58.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:57.805+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:58.549 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:58.234+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:58.550 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:58.234+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:58.550 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:58.234+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:58.550 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:58.235+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:58.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:58.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:58.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:58.548+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:59.281 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:58.848+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:59.282 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:58.848+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:59.282 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:58.848+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:54:59.282 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:58.848+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:54:59.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:59.528+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:59.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:59.528+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:59.528+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:54:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:54:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:54:59.529+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:54:59.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:59.279+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:59.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:59.279+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:59.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:59.279+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:54:59.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:54:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:54:59.280+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:00.257 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:59.854+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:00.257 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:59.854+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:00.258 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:59.854+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:00.258 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:54:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:54:59.855+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:00.533 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:00.293+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:00.533 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:00.293+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:00.534 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:00.293+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:00.534 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:00.294+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:00.532+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:00.532+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:00.532+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:00.533+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:01.321 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:00.876+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:01.321 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:00.876+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:01.321 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:00.876+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:01.321 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:00.876+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:01.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:01.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:01.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:01.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:01.551+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:01.319+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:01.319+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:01.319+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:01.320+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:01.808 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.806+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:55:01.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.806+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:55:01.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.806+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f17e8064000 tx=0x7f17e800f050 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:55:01.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.806+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f17e8064000 tx=0x7f17e800f050 comp rx=0 tx=0).stop 2024-09-10T03:55:01.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.806+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d8048c70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:55:01.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.806+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d8048c70 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:01.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.806+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:01.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.807+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:01.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.807+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047f60 con 0x7f17d805a5c0 2024-09-10T03:55:01.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.807+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d80486a0 con 0x7f17f0101a40 2024-09-10T03:55:01.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.807+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:55:01.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.808+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:55:01.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.808+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:01.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.808+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048aa0 con 0x7f17d805a5c0 2024-09-10T03:55:01.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.808+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f17e003f250 tx=0x7f17e8035600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:55:01.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.809+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0006be0 con 0x7f17d805a5c0 2024-09-10T03:55:01.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.809+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e005eb90 con 0x7f17d805a5c0 2024-09-10T03:55:01.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:01.810+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d805a5c0 2024-09-10T03:55:02.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:55:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:55:01.807+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6147 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:55:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:01.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:01.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:01.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:01.874+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:02.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:02.518+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:02.518+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:02.518+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:02.519+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:02.345+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:02.345+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:02.345+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:02.346+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:03.316 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:02.865+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:03.317 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:02.865+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:03.317 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:02.865+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:03.317 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:02.866+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:03.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:03.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:03.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:03.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:03.480+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:03.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:03.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:03.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:03.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:03.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:03.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:03.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:03.316+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:04.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:03.907+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:04.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:03.907+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:03.907+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:03.907+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:04.465+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:04.465+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:04.465+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:04.466+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:04.359+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:04.359+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:04.359+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:04.360+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:05.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:04.950+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:04.950+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:04.950+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:04.951+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:05.568 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:05.567+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:55:05.568 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:05.567+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:55:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:05.516+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:05.516+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:05.516+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:05.516+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:05.379+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:05.379+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:05.379+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:05.380+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:06.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:05.905+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:06.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:05.905+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:06.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:05.905+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:06.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:05.906+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:06.809 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.807+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:55:06.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.807+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:55:06.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.807+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f17e003f250 tx=0x7f17e8035600 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:55:06.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.808+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f17e003f250 tx=0x7f17e8035600 comp rx=0 tx=0).stop 2024-09-10T03:55:06.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.808+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:55:06.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.808+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:06.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.808+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:06.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.808+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:06.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.808+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047f60 con 0x7f17d8047ae0 2024-09-10T03:55:06.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.808+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:55:06.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:06.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:06.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:06.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:06.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:06.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:06.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:06.414+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:06.811 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:06.512+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:06.811 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:06.512+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:06.811 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:06.512+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:06.812 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:06.513+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:06.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.808+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:55:06.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.809+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:55:06.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.809+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:06.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.809+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048170 con 0x7f17d8047ae0 2024-09-10T03:55:06.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.809+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f17d8018990 tx=0x7f17e0050fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:55:06.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.810+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028050 con 0x7f17d8047ae0 2024-09-10T03:55:06.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.810+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e00153b0 con 0x7f17d8047ae0 2024-09-10T03:55:06.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:06.811+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d8047ae0 2024-09-10T03:55:07.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:55:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:55:06.808+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6204 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:55:07.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:06.894+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:06.894+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:06.894+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:06.895+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:07.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:07.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:07.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:07.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:07.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:07.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:07.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:07.371+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:07.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:07.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:07.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:07.550+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:08.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:07.878+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:08.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:07.878+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:07.878+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:07.879+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:08.641 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:08.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:08.641 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:08.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:08.641 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:08.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:08.641 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:08.592+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:08.642 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:08.364+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:08.642 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:08.364+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:08.642 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:08.364+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:08.642 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:08.365+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:09.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:08.857+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:09.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:08.857+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:09.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:08.857+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:09.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:08.858+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:09.603 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:09.347+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:09.604 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:09.347+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:09.604 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:09.347+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:09.604 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:09.348+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:10.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:09.845+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:09.845+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:09.845+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:09.846+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:09.602+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:09.602+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:10.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:09.602+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:10.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:09.603+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:10.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:10.344+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:10.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:10.344+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:10.344+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:10.345+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:10.579+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:10.579+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:10.579+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:10.580+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:11.309 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:11.308+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:11.310 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:11.308+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:11.310 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:11.308+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:11.310 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:10.876+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:11.310 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:10.876+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:11.310 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:10.876+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:11.310 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:10.877+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:11.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:11.308+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:11.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:11.544+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:11.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:11.544+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:11.544+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:11.544+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:11.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.808+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:55:11.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.808+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:55:11.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.808+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f17d8018990 tx=0x7f17e0050fc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:55:11.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.808+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f17d8018990 tx=0x7f17e0050fc0 comp rx=0 tx=0).stop 2024-09-10T03:55:11.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.808+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:55:11.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.808+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:11.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.808+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:11.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.808+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:11.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.808+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047f60 con 0x7f17d805a5c0 2024-09-10T03:55:11.810 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.808+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:55:11.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.809+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:55:11.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.809+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:55:11.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.809+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:11.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.809+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048170 con 0x7f17d805a5c0 2024-09-10T03:55:11.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.810+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f17e8013f80 tx=0x7f17e0015710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:55:11.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.810+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055040 con 0x7f17d805a5c0 2024-09-10T03:55:11.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.810+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8039e70 con 0x7f17d805a5c0 2024-09-10T03:55:11.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:11.811+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c040 con 0x7f17d805a5c0 2024-09-10T03:55:12.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:55:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:55:11.809+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6258 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:55:12.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:11.920+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:11.920+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:11.920+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:11.921+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:12.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:12.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:12.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:12.575+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:12.352+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:12.352+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:12.352+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:12.352+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:13.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:12.921+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:13.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:12.921+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:13.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:12.921+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:13.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:12.922+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:13.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:13.396+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:13.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:13.396+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:13.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:13.396+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:13.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:13.396+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:13.556+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:13.556+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:13.556+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:13.556+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:14.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:13.901+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:14.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:13.901+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:14.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:13.901+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:14.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:13.902+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:14.784 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:14.446+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:14.785 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:14.446+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:14.785 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:14.446+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:14.785 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:14.447+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:14.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:14.537+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:14.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:14.537+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:14.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:14.537+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:14.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:14.537+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:15.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:14.887+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:15.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:14.887+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:15.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:14.887+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:15.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:14.888+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:15.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:15.472+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:15.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:15.472+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:15.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:15.472+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:15.473+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:15.534+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:15.534+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:15.534+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:15.535+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:16.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:15.883+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:16.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:15.883+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:15.883+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:15.884+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:16.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.809+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:55:16.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.809+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:55:16.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.809+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f17e8013f80 tx=0x7f17e0015710 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:55:16.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.809+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f17e8013f80 tx=0x7f17e0015710 comp rx=0 tx=0).stop 2024-09-10T03:55:16.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.809+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:55:16.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.809+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:16.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.809+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:16.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.809+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:16.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.809+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047f60 con 0x7f17d8047ae0 2024-09-10T03:55:16.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.809+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:55:16.811 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.810+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:55:16.812 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:16.557+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:16.812 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:16.557+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:16.812 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:16.557+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:16.812 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:16.558+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:16.812 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:16.469+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:16.812 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:16.469+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:16.812 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:16.469+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:16.812 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:16.470+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:16.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.810+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:55:16.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.810+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:16.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.811+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048d60 con 0x7f17d8047ae0 2024-09-10T03:55:16.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.811+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f17e800c010 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:55:16.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.812+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8014df0 con 0x7f17d8047ae0 2024-09-10T03:55:16.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.812+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d8047ae0 2024-09-10T03:55:16.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:16.813+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c040 con 0x7f17d8047ae0 2024-09-10T03:55:17.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:55:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:55:16.810+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6312 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:55:17.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:16.918+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:17.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:16.918+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:17.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:16.918+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:17.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:16.918+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:17.529+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:17.529+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:17.529+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:17.530+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:17.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:17.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:17.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:17.512+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:18.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:17.935+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:18.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:17.935+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:18.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:17.935+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:18.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:17.935+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:18.526+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:18.526+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:18.526+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:18.527+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:18.553+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:18.553+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:18.553+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:18.554+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:19.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:18.985+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:19.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:18.985+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:19.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:18.985+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:19.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:18.985+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:19.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:19.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:19.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:19.493+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:19.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:19.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:19.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:19.527+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:20.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:19.935+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:19.935+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:19.935+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:19.936+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:20.500+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:20.500+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:20.500+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:20.501+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:20.493+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:20.493+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:20.493+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:20.494+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:20.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:20.927+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:55:20.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:20.927+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:55:21.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:20.966+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:21.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:20.966+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:21.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:20.966+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:21.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:20.966+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:21.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.810+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:55:21.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.810+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:55:21.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.810+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f17e800c010 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:55:21.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.810+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f17e800c010 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:55:21.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.810+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:55:21.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.810+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:21.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.810+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:21.812 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.810+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:21.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.810+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8055670 con 0x7f17d805a5c0 2024-09-10T03:55:21.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.810+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047f60 con 0x7f17f0101a40 2024-09-10T03:55:21.813 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:21.503+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:21.813 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:21.503+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:21.813 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:21.503+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:21.813 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:21.504+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:21.813 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:21.542+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:21.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:21.542+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:21.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:21.542+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:21.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:21.542+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:21.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.811+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:55:21.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.812+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:55:21.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.812+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:21.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.812+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048be0 con 0x7f17d805a5c0 2024-09-10T03:55:21.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.812+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f17e8056eb0 tx=0x7f17e8028e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:55:21.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.816+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055040 con 0x7f17d805a5c0 2024-09-10T03:55:21.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.816+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d805a5c0 2024-09-10T03:55:21.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:21.817+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c040 con 0x7f17d805a5c0 2024-09-10T03:55:22.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:55:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:55:21.811+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6369 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:55:22.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:21.999+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:22.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:21.999+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:22.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:21.999+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:22.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:21.999+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:22.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:22.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:22.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:22.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:22.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:22.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:22.550+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:22.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:22.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:22.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:22.550+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:23.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:22.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:23.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:22.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:23.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:22.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:23.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:22.999+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:23.569+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:23.569+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:23.569+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:23.570+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:23.551+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:23.551+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:23.551+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:23.552+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:24.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:24.014+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:24.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:24.014+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:24.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:24.014+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:24.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:24.015+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:24.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:24.574+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:24.574+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:24.574+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:24.574+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:24.604+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:24.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:24.604+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:24.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:24.604+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:24.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:24.605+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:25.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:25.028+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:25.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:25.028+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:25.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:25.028+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:25.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:25.028+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:25.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:25.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:25.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:25.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:25.625+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:25.560+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:25.560+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:25.560+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:25.561+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:26.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:26.024+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:26.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:26.024+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:26.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:26.024+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:26.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:26.025+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:26.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.811+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:55:26.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.811+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:55:26.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.811+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f17e8056eb0 tx=0x7f17e8028e50 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:55:26.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.811+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f17e8056eb0 tx=0x7f17e8028e50 comp rx=0 tx=0).stop 2024-09-10T03:55:26.813 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.812+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:55:26.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.812+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:26.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.812+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:26.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.812+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:26.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.812+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047f60 con 0x7f17d8047ae0 2024-09-10T03:55:26.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.812+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:55:26.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.812+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:55:26.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.812+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:55:26.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.812+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:26.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.812+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048d60 con 0x7f17d8047ae0 2024-09-10T03:55:26.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.812+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f17e800b770 tx=0x7f17e801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:55:26.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.812+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055050 con 0x7f17d8047ae0 2024-09-10T03:55:26.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.812+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d8047ae0 2024-09-10T03:55:26.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:26.812+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064070 con 0x7f17d8047ae0 2024-09-10T03:55:27.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:26.588+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:27.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:26.588+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:27.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:26.588+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:27.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:26.589+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:27.075 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:55:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:55:26.812+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6429 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:55:27.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:26.645+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:27.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:26.645+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:27.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:26.645+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:27.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:26.645+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:27.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:27.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:27.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:27.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:27.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:27.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:27.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:27.074+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:28.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:27.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:27.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:27.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:27.597+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:27.629+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:27.629+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:27.629+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:27.630+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:28.550 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:28.087+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:28.550 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:28.087+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:28.550 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:28.087+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:28.550 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:28.088+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:28.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:28.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:28.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:28.549+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:28.671+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:28.671+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:28.671+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:28.672+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:29.578 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:29.098+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:29.578 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:29.098+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:29.578 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:29.098+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:29.579 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:29.099+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:29.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:29.577+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:29.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:29.577+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:29.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:29.577+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:29.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:29.578+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:29.623+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:29.623+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:29.623+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:29.624+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:30.553 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:30.111+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:30.553 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:30.111+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:30.553 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:30.111+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:30.553 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:30.112+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:30.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:30.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:30.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:30.552+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:30.666+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:30.666+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:30.666+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:30.667+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:31.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:31.067+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:31.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:31.067+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:31.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:31.067+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:31.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:31.067+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:31.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.811+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:55:31.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.811+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:55:31.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.811+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f17e800b770 tx=0x7f17e801e000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:55:31.814 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.811+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f17e800b770 tx=0x7f17e801e000 comp rx=0 tx=0).stop 2024-09-10T03:55:31.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.811+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:55:31.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.811+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:31.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.811+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:31.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.811+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:31.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.811+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047f60 con 0x7f17d805a5c0 2024-09-10T03:55:31.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.811+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:55:31.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.812+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:55:31.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.812+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:55:31.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.812+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:31.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.813+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048d60 con 0x7f17d805a5c0 2024-09-10T03:55:31.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.813+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f17e0020f90 tx=0x7f17e803fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:55:31.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.814+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c040 con 0x7f17d805a5c0 2024-09-10T03:55:31.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.814+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0034720 con 0x7f17d805a5c0 2024-09-10T03:55:31.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:31.815+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e003c8c0 con 0x7f17d805a5c0 2024-09-10T03:55:32.077 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:31.582+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:32.078 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:31.582+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:32.078 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:31.582+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:32.078 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:31.583+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:32.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:31.645+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:32.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:31.645+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:32.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:31.645+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:32.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:31.646+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:32.079 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:55:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:55:31.812+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6489 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:55:32.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:32.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:32.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:32.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:32.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:32.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:32.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:32.076+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:33.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:32.600+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:33.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:32.600+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:33.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:32.600+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:33.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:32.600+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:33.035 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:32.600+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:33.035 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:32.600+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:33.035 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:32.600+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:33.035 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:32.600+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:33.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:33.032+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:33.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:33.032+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:33.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:33.032+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:33.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:33.032+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:33.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:33.636+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:33.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:33.636+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:33.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:33.636+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:33.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:33.637+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:33.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:33.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:33.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:33.626+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:34.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:33.995+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:34.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:33.995+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:34.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:33.995+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:34.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:33.996+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:34.959 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:34.666+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:34.960 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:34.666+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:34.960 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:34.666+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:34.960 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:34.667+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:34.960 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:34.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:34.960 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:34.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:34.960 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:34.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:34.960 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:34.626+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:35.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:34.958+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:35.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:34.958+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:35.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:34.958+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:35.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:34.958+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:35.925 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:35.665+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:35.925 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:35.665+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:35.925 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:35.665+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:35.925 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:35.665+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:35.926 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:35.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:35.926 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:35.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:35.926 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:35.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:35.926 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:35.597+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:36.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.287+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:55:36.288 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.287+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:55:36.289 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:35.923+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:36.289 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:35.923+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:36.289 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:35.923+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:36.289 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:35.924+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:36.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.812+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:55:36.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.812+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:55:36.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.812+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f17e0020f90 tx=0x7f17e803fbe0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:55:36.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.812+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f17e0020f90 tx=0x7f17e803fbe0 comp rx=0 tx=0).stop 2024-09-10T03:55:36.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.812+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:55:36.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.812+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:36.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.812+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:36.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.812+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:36.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.812+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17d8047ae0 2024-09-10T03:55:36.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.812+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8047f60 con 0x7f17f0101a40 2024-09-10T03:55:36.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.813+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:55:36.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.813+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:55:36.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.813+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:36.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.813+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048d60 con 0x7f17d8047ae0 2024-09-10T03:55:36.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.814+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f17e803e050 tx=0x7f17e001a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:55:36.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.814+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055070 con 0x7f17d8047ae0 2024-09-10T03:55:36.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.816+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b050 con 0x7f17d8047ae0 2024-09-10T03:55:36.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:36.816+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17d8047ae0 2024-09-10T03:55:36.926 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:36.583+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:36.926 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:36.583+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:36.926 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:36.583+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:36.926 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:36.583+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:36.926 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:36.709+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:36.926 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:36.709+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:36.926 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:36.709+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:36.926 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:36.709+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:36.927 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:55:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:55:36.813+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6552 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:55:37.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:36.923+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:37.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:36.923+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:37.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:36.923+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:37.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:36.923+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:37.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:37.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:37.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:37.552+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:37.745+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:37.745+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:37.745+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:37.746+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:38.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:37.875+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:38.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:37.875+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:38.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:37.875+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:38.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:37.876+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:38.922 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:38.592+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:38.922 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:38.592+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:38.923 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:38.592+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:38.923 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:38.593+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:38.923 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:38.795+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:38.923 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:38.795+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:38.923 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:38.795+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:38.923 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:38.796+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:39.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:38.921+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:39.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:38.921+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:39.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:38.921+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:39.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:38.922+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:39.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:39.555+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:39.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:39.555+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:39.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:39.555+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:39.556+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:40.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:39.950+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:40.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:39.950+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:40.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:39.950+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:40.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:39.951+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:39.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:39.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:39.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:39.846+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:40.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:40.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:40.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:40.528+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:41.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:40.924+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:41.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:40.924+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:41.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:40.924+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:41.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:40.924+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:40.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:40.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:40.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:40.870+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:41.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.814+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:55:41.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.814+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:55:41.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.814+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f17e803e050 tx=0x7f17e001a8c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:55:41.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.814+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f17e803e050 tx=0x7f17e001a8c0 comp rx=0 tx=0).stop 2024-09-10T03:55:41.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.814+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:55:41.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.814+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:41.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.814+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:41.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.814+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:41.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.814+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17d805a5c0 2024-09-10T03:55:41.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.814+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048d60 con 0x7f17f0101a40 2024-09-10T03:55:41.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.815+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:55:41.818 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:41.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:41.818 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:41.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:41.818 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:41.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:41.818 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:41.552+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:41.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.816+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:55:41.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.816+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:41.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.816+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048ee0 con 0x7f17d805a5c0 2024-09-10T03:55:41.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.816+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f17d801a320 tx=0x7f17e8055000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:55:41.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.818+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028070 con 0x7f17d805a5c0 2024-09-10T03:55:41.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.818+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0047d80 con 0x7f17d805a5c0 2024-09-10T03:55:41.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:41.819+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d805a5c0 2024-09-10T03:55:42.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:41.841+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:42.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:41.841+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:42.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:41.841+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:42.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:41.842+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:41.924+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:41.924+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:41.924+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:41.924+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:42.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:55:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:55:41.815+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6615 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:55:42.808 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:42.521+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:42.808 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:42.521+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:42.808 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:42.521+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:42.808 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:42.522+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:43.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:42.806+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:42.806+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:42.806+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:42.807+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:42.887+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:42.887+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:42.887+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:42.887+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:43.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:43.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:43.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:43.568+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:43.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:43.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:43.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:43.853+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:44.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:43.899+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:44.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:43.899+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:44.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:43.899+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:44.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:43.899+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:44.572+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:44.572+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:44.572+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:44.573+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:44.850+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:44.850+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:44.850+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:44.851+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:44.871+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:44.871+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:44.871+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:44.872+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:45.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:45.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:45.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:45.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:45.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:45.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:45.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:45.550+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:46.209 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:45.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:46.209 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:45.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:46.209 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:45.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:46.210 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:45.833+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:46.210 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:45.841+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:46.210 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:45.841+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:46.210 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:45.841+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:46.210 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:45.842+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:46.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.816+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:55:46.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.816+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:55:46.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.816+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f17d801a320 tx=0x7f17e8055000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:55:46.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.816+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f17d801a320 tx=0x7f17e8055000 comp rx=0 tx=0).stop 2024-09-10T03:55:46.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.816+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:55:46.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.816+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:46.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.816+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:46.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.816+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:46.818 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:46.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:46.818 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:46.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:46.819 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:46.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:46.819 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:46.793+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:46.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.816+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17d8047ae0 2024-09-10T03:55:46.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.816+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048d60 con 0x7f17f0101a40 2024-09-10T03:55:46.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.816+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:55:46.820 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:46.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:46.820 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:46.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:46.820 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:46.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:46.820 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:46.784+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:46.820 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:46.562+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:46.820 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:46.562+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:46.820 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:46.562+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:46.821 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:46.563+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:46.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.817+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:55:46.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.817+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:46.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.817+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80490b0 con 0x7f17d8047ae0 2024-09-10T03:55:46.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.817+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f17e80445c0 tx=0x7f17e000cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:55:46.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.818+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e800d6e0 con 0x7f17d8047ae0 2024-09-10T03:55:46.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.818+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8014de0 con 0x7f17d8047ae0 2024-09-10T03:55:46.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:46.820+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17d8047ae0 2024-09-10T03:55:47.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:55:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:55:46.816+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6678 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:55:47.789 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:47.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:47.789 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:47.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:47.789 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:47.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:47.789 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:47.528+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:48.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:47.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:48.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:47.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:47.788+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:47.788+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:47.834+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:47.834+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:47.834+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:47.834+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:48.828 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:48.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:48.828 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:48.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:48.828 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:48.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:48.828 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:48.550+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:48.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:48.757+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:48.829 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:48.757+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:48.829 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:48.757+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:48.829 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:48.758+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:49.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:48.826+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:49.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:48.826+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:49.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:48.826+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:48.827+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:49.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:49.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:49.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:49.575+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:49.723+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:49.723+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:49.723+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:49.724+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:49.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:49.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:49.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:49.864+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:50.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:50.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:50.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:50.549+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:50.723+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:50.723+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:50.723+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:50.723+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:51.316 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:50.877+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:51.316 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:50.877+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:51.316 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:50.877+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:51.316 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:50.878+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:51.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:51.500+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:51.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:51.500+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:51.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:51.500+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:51.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:51.501+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:51.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.647+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:55:51.648 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.647+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:55:51.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.817+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:55:51.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.817+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:55:51.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.817+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f17e80445c0 tx=0x7f17e000cab0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:55:51.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.817+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f17e80445c0 tx=0x7f17e000cab0 comp rx=0 tx=0).stop 2024-09-10T03:55:51.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.818+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:55:51.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.818+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:51.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.818+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:51.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.818+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:51.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.818+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17d805a5c0 2024-09-10T03:55:51.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.818+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048d60 con 0x7f17f0101a40 2024-09-10T03:55:51.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.818+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:55:51.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.819+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:55:51.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.819+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:51.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.819+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80490b0 con 0x7f17d805a5c0 2024-09-10T03:55:51.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.819+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f17d801a320 tx=0x7f17e8029d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:55:51.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.819+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e80445f0 con 0x7f17d805a5c0 2024-09-10T03:55:51.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.819+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b050 con 0x7f17d805a5c0 2024-09-10T03:55:51.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:51.820+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c070 con 0x7f17d805a5c0 2024-09-10T03:55:51.903 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:55:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:55:51.817+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6740 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:55:51.904 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:51.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:51.904 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:51.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:51.904 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:51.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:51.904 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:51.700+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:52.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:51.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:52.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:51.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:52.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:51.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:52.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:51.903+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:52.533+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:52.533+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:52.533+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:52.534+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:52.683+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:52.683+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:52.683+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:52.683+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:52.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:52.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:52.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:52.932+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:53.495+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:53.495+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:53.495+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:53.496+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:53.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:53.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:53.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:53.665+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:53.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:53.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:53.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:53.890+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:54.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:54.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:54.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:54.479+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:54.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:54.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:54.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:54.686+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:54.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:54.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:54.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:54.898+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:55.498+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:55.498+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:55.498+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:55.499+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:55.687+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:55.687+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:55.687+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:55.688+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:55.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:55.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:55.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:55.907+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:56.698 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:56.496+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:56.698 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:56.496+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:56.699 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:56.496+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:56.699 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:56.497+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:56.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.818+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:55:56.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.818+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:55:56.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.818+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f17d801a320 tx=0x7f17e8029d80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:55:56.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.818+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f17d801a320 tx=0x7f17e8029d80 comp rx=0 tx=0).stop 2024-09-10T03:55:56.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.818+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:55:56.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.818+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:56.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.818+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:56.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.818+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:55:56.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.818+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17d8047ae0 2024-09-10T03:55:56.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.818+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048d60 con 0x7f17f0101a40 2024-09-10T03:55:56.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.818+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:55:56.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.819+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:55:56.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.819+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:55:56.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.820+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048170 con 0x7f17d8047ae0 2024-09-10T03:55:56.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.820+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f17e0020ee0 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:55:56.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.822+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c040 con 0x7f17d8047ae0 2024-09-10T03:55:56.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.822+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0047d80 con 0x7f17d8047ae0 2024-09-10T03:55:56.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:55:56.822+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8047ae0 2024-09-10T03:55:56.949 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:55:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:55:56.818+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6803 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:55:56.949 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:56.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:56.949 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:56.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:56.950 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:56.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:56.950 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:56.698+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:56.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:56.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:56.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:56.948+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:57.793 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:57.665+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:57.793 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:57.665+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:57.793 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:57.665+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:57.793 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:57.666+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:57.799 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:57.471+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:57.799 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:57.471+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:57.799 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:57.471+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:57.799 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:57.472+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:57.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:57.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:57.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:57.942+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:58.437+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:58.437+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:58.437+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:58.438+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:58.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:58.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:58.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:58.685+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:55:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:58.892+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:58.892+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:58.892+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:55:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:58.892+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:59.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:59.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:59.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:59.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:59.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:59.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:55:59.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:55:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:55:59.451+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:55:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:59.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:59.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:59.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:55:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:55:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:55:59.653+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:59.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:59.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:59.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:55:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:55:59.889+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:00.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:00.621+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:00.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:00.621+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:00.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:00.621+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:00.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:00.622+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:00.425+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:00.425+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:00.425+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:00.425+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:00.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:00.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:00.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:00.845+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:01.664 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:01.379+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:01.664 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:01.379+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:01.664 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:01.379+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:01.665 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:01.380+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:01.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.819+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:56:01.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.819+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:56:01.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.819+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f17e0020ee0 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:56:01.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.819+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f17e0020ee0 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:56:01.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.819+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:56:01.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.819+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:01.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.819+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:01.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.819+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:01.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.819+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d804f4e0 con 0x7f17d805a5c0 2024-09-10T03:56:01.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.819+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17f0101a40 2024-09-10T03:56:01.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.820+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:56:01.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.820+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:56:01.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.820+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:01.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.820+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048ee0 con 0x7f17d805a5c0 2024-09-10T03:56:01.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.821+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f17e8058f20 tx=0x7f17e0011860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:56:01.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.821+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e805c430 con 0x7f17d805a5c0 2024-09-10T03:56:01.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.821+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d805a5c0 2024-09-10T03:56:01.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:01.823+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d805a5c0 2024-09-10T03:56:02.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:56:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:56:01.820+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6866 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:56:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:01.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:01.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:01.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:01.663+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:01.886+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:01.886+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:01.886+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:01.886+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:02.644 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:02.365+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:02.644 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:02.365+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:02.644 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:02.365+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:02.644 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:02.366+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:03.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:02.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:03.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:02.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:03.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:02.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:02.880+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:02.642+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:02.642+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:02.642+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:02.643+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:03.639 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:03.333+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:03.639 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:03.333+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:03.639 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:03.333+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:03.639 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:03.334+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:03.906 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:03.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:03.907 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:03.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:03.907 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:03.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:03.907 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:03.638+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:04.304 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:03.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:04.305 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:03.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:04.305 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:03.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:04.305 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:03.906+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:04.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:04.303+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:04.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:04.303+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:04.303+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:04.304+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:04.926 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:04.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:04.927 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:04.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:04.927 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:04.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:04.927 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:04.682+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:04.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:04.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:04.925+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:04.926+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:05.670 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:05.342+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:05.670 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:05.342+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:05.670 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:05.342+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:05.670 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:05.343+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:06.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:05.668+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:06.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:05.668+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:06.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:05.668+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:05.669+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:05.903+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:05.903+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:05.903+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:05.904+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:06.620 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:06.330+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:06.620 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:06.330+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:06.620 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:06.330+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:06.620 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:06.331+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:06.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.820+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:56:06.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.820+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:56:06.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.820+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f17e8058f20 tx=0x7f17e0011860 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:56:06.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.820+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f17e8058f20 tx=0x7f17e0011860 comp rx=0 tx=0).stop 2024-09-10T03:56:06.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.820+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:56:06.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.820+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:06.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.820+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:06.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.820+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:06.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.820+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17d8047ae0 2024-09-10T03:56:06.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.820+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:56:06.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.820+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:56:06.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.821+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:56:06.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.821+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:06.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.821+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80490b0 con 0x7f17d8047ae0 2024-09-10T03:56:06.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.821+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f17e0014f30 tx=0x7f17e800f0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:56:06.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.822+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c070 con 0x7f17d8047ae0 2024-09-10T03:56:06.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.822+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e00419e0 con 0x7f17d8047ae0 2024-09-10T03:56:06.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:06.823+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8047ae0 2024-09-10T03:56:06.923 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:56:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:56:06.821+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6929 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:56:06.924 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:06.652+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:06.924 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:06.652+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:06.924 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:06.652+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:06.924 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:06.653+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:07.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:07.263+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:56:07.264 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:07.263+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:56:07.265 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:06.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:07.265 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:06.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:07.265 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:06.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:07.265 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:06.923+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:07.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:07.376+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:07.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:07.376+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:07.376+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:07.377+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:07.971 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:07.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:07.971 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:07.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:07.971 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:07.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:07.971 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:07.637+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:07.969+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:07.969+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:07.969+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:08.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:07.970+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:08.663 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:08.357+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:08.663 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:08.357+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:08.663 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:08.357+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:08.663 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:08.357+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:09.021 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:08.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:09.021 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:08.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:09.021 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:08.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:09.021 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:08.661+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:09.019+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:09.019+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:09.019+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:09.019+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:09.641 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:09.358+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:09.641 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:09.358+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:09.641 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:09.358+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:09.641 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:09.359+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:10.071 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:09.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:10.071 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:09.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:10.071 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:09.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:10.071 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:09.639+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:10.069+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:10.069+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:10.069+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:10.069+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:10.683 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:10.354+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:10.683 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:10.354+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:10.683 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:10.354+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:10.683 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:10.355+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:11.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:10.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:11.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:10.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:11.058 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:10.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:11.058 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:10.681+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:11.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:11.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:11.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:11.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:11.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:11.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:11.055+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:11.648 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:11.384+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:11.648 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:11.384+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:11.648 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:11.384+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:11.648 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:11.385+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:11.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.820+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T03:56:11.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.820+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:56:11.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.820+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f17e0014f30 tx=0x7f17e800f0b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:56:11.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.820+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f17e0014f30 tx=0x7f17e800f0b0 comp rx=0 tx=0).stop 2024-09-10T03:56:11.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.820+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:56:11.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.820+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:11.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.820+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:11.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.820+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:11.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.820+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17d805a5c0 2024-09-10T03:56:11.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.820+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:56:11.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.821+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:56:11.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.821+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:56:11.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.821+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:11.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.821+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80490b0 con 0x7f17d805a5c0 2024-09-10T03:56:11.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.822+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f17e8065f00 tx=0x7f17e00117f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:56:11.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.822+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055050 con 0x7f17d805a5c0 2024-09-10T03:56:11.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.823+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d805a5c0 2024-09-10T03:56:11.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:11.825+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8026390 con 0x7f17d805a5c0 2024-09-10T03:56:12.053 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:56:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:56:11.821+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 6992 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:56:12.285 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:11.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:12.285 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:11.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:12.285 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:11.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:12.286 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:11.646+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:12.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:12.050+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:12.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:12.050+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:12.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:12.050+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:12.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:12.051+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:12.411+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:12.411+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:12.411+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:12.411+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:12.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:12.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:12.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:12.600+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:13.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:13.057+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:13.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:13.057+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:13.057+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:13.058+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:13.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:13.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:13.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:13.622+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:13.384+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:13.384+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:13.384+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:13.385+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:14.024+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:14.024+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:14.024+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:14.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:14.024+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:14.663 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:14.384+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:14.663 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:14.384+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:14.663 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:14.384+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:14.663 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:14.384+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:15.072 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:14.662+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:15.072 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:14.662+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:15.072 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:14.662+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:15.072 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:14.662+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:15.071+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:15.071+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:15.071+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:15.072+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:15.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:15.410+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:15.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:15.410+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:15.410+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:15.411+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:15.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:15.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:15.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:15.654+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:16.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:16.073+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:16.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:16.073+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:16.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:16.073+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:16.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:16.074+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:16.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.822+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:56:16.823 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.822+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:56:16.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.822+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f17e8065f00 tx=0x7f17e00117f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:56:16.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.822+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f17e8065f00 tx=0x7f17e00117f0 comp rx=0 tx=0).stop 2024-09-10T03:56:16.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.822+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:56:16.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.822+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:16.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.822+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:16.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.822+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:16.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.822+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17d8047ae0 2024-09-10T03:56:16.824 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.822+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:56:16.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:16.652+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:16.825 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:16.652+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:16.825 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:16.652+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:16.825 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:16.652+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:16.825 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:16.447+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:16.825 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:16.447+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:16.825 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:16.447+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:16.825 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:16.448+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:16.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.823+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:56:16.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.823+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:56:16.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.823+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:16.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.823+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80490b0 con 0x7f17d8047ae0 2024-09-10T03:56:16.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.823+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f17e8065ed0 tx=0x7f17e800bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:56:16.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.824+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e804ec30 con 0x7f17d8047ae0 2024-09-10T03:56:16.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.824+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d8047ae0 2024-09-10T03:56:16.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:16.825+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d8047ae0 2024-09-10T03:56:17.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:17.028+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:17.028+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:17.028+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:17.028+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:17.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:56:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:56:16.823+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 7055 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:56:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:17.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:17.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:17.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:17.488+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:17.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:17.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:17.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:17.699+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:18.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:18.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:18.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:18.032+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:18.518+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:18.518+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:18.518+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:18.519+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:18.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:18.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:18.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:18.682+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:19.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:19.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:19.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:19.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:19.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:19.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:19.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:19.076+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:19.482+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:19.482+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:19.482+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:19.482+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:19.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:19.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:19.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:19.634+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:20.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:20.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:20.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:20.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:20.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:20.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:20.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:20.030+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:20.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:20.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:20.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:20.451+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:20.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:20.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:20.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:20.598+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:21.045+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:21.045+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:21.045+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:21.046+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:21.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.823+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:56:21.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.823+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:56:21.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.823+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f17e8065ed0 tx=0x7f17e800bc00 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:56:21.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.823+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f17e8065ed0 tx=0x7f17e800bc00 comp rx=0 tx=0).stop 2024-09-10T03:56:21.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.823+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:56:21.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.823+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:21.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.823+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:21.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.823+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:21.825 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.823+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17d805a5c0 2024-09-10T03:56:21.826 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:21.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:21.826 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:21.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:21.826 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:21.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:21.826 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:21.556+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:21.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.823+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:56:21.826 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:21.458+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:21.827 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:21.458+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:21.827 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:21.458+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:21.827 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:21.459+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:21.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.824+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:56:21.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.824+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:56:21.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.824+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:21.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.824+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80490b0 con 0x7f17d805a5c0 2024-09-10T03:56:21.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.825+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f17d804f710 tx=0x7f17e8009d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:56:21.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.826+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e804e8c0 con 0x7f17d805a5c0 2024-09-10T03:56:21.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.826+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e801eaf0 con 0x7f17d805a5c0 2024-09-10T03:56:21.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:21.826+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d805a5c0 2024-09-10T03:56:22.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:22.019+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:22.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:22.019+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:22.019+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:22.021+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:22.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:56:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:56:21.824+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 7118 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:56:22.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:22.623+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:56:22.624 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:22.623+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:56:22.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:22.416+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:22.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:22.416+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:22.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:22.416+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:22.417+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:22.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:22.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:22.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:22.549+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:23.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:23.063+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:23.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:23.063+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:23.063+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:23.064+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:23.415+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:23.415+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:23.415+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:23.416+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:23.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:23.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:23.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:23.561+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:24.419 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:24.104+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:24.419 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:24.104+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:24.419 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:24.104+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:24.419 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:24.105+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:24.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:24.418+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:24.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:24.418+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:24.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:24.418+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:24.418+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:24.540+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:24.540+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:24.540+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:24.541+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:25.433 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:25.132+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:25.433 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:25.132+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:25.433 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:25.132+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:25.433 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:25.133+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:25.696 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:25.432+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:25.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:25.432+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:25.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:25.432+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:25.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:25.432+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:25.697 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:25.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:25.697 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:25.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:25.697 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:25.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:25.697 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:25.587+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:26.563 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:26.388+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:26.563 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:26.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:26.563 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:26.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:26.563 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:26.389+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:26.564 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:26.159+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:26.564 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:26.159+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:26.564 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:26.159+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:26.564 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:26.159+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:26.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.824+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:56:26.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.824+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:56:26.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.824+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f17d804f710 tx=0x7f17e8009d70 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:56:26.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.824+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f17d804f710 tx=0x7f17e8009d70 comp rx=0 tx=0).stop 2024-09-10T03:56:26.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.824+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:56:26.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.824+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:26.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.824+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:26.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.824+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:26.826 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.824+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17d8047ae0 2024-09-10T03:56:26.827 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.824+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:56:26.827 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:26.562+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:26.827 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:26.562+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:26.827 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:26.562+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:26.827 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:26.562+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:26.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.825+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:56:26.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.826+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:56:26.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.826+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:26.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.826+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048ee0 con 0x7f17d8047ae0 2024-09-10T03:56:26.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.826+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f17d800fce0 tx=0x7f17e805bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:56:26.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.828+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e805cbd0 con 0x7f17d8047ae0 2024-09-10T03:56:26.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.828+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8022ed0 con 0x7f17d8047ae0 2024-09-10T03:56:26.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:26.829+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d8047ae0 2024-09-10T03:56:27.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:56:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:56:26.825+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 7181 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:56:27.399 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:27.176+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:27.399 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:27.176+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:27.399 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:27.176+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:27.399 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:27.177+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:27.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:27.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:27.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:27.536+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:27.397+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:27.397+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:27.397+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:27.398+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:28.563 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:28.381+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:28.563 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:28.381+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:28.563 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:28.381+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:28.563 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:28.381+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:28.563 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:28.190+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:28.563 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:28.190+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:28.564 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:28.190+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:28.564 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:28.191+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:28.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:28.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:28.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:28.562+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:29.342+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:29.342+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:29.342+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:29.343+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:29.222+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:29.222+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:29.222+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:29.223+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:30.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:29.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:30.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:29.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:29.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:29.607+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:30.322+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:30.322+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:30.322+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:30.323+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:30.193+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:30.193+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:30.193+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:30.194+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:31.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:30.604+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:31.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:30.604+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:30.604+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:30.605+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:31.557 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:31.360+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:31.558 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:31.360+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:31.558 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:31.360+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:31.558 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:31.361+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:31.558 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:31.183+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:31.558 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:31.183+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:31.558 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:31.183+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:31.558 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:31.184+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:31.827 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:31.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:31.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:31.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:31.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:31.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:31.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:31.557+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:31.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.826+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:56:31.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.826+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:56:31.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.826+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f17d800fce0 tx=0x7f17e805bc90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:56:31.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.826+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f17d800fce0 tx=0x7f17e805bc90 comp rx=0 tx=0).stop 2024-09-10T03:56:31.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.826+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:56:31.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.826+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:31.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.826+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:31.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.826+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:31.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.826+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17d805a5c0 2024-09-10T03:56:31.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.826+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:56:31.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.826+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:56:31.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.827+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:56:31.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.827+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:31.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.827+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80490b0 con 0x7f17d805a5c0 2024-09-10T03:56:31.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.827+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f17d8007550 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:56:31.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.827+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8049220 con 0x7f17d805a5c0 2024-09-10T03:56:31.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.828+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e8063df0 con 0x7f17d805a5c0 2024-09-10T03:56:31.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:31.829+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8005d10 con 0x7f17d805a5c0 2024-09-10T03:56:32.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:56:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:56:31.826+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 7244 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:56:32.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:32.318+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:32.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:32.318+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:32.318+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:32.319+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:32.168+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:32.168+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:32.168+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:32.169+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:33.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:32.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:33.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:32.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:32.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:32.592+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:33.568 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:33.329+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:33.568 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:33.329+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:33.568 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:33.329+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:33.568 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:33.330+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:33.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:33.214+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:33.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:33.214+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:33.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:33.214+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:33.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:33.215+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:33.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:33.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:33.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:33.593+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:34.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:34.320+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:34.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:34.320+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:34.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:34.320+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:34.321+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:34.191+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:34.191+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:34.191+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:34.192+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:34.912 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:34.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:34.912 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:34.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:34.912 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:34.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:34.912 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:34.625+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:35.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:35.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:35.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:35.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:35.163+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:35.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:35.282+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:35.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:35.282+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:35.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:35.282+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:35.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:35.283+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:36.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:35.606+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:36.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:35.606+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:36.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:35.606+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:36.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:35.607+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:36.566 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:36.290+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:36.566 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:36.290+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:36.566 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:36.290+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:36.566 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:36.290+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:36.566 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:36.166+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:36.566 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:36.166+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:36.567 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:36.166+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:36.567 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:36.167+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:36.828 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.827+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:56:36.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.827+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:56:36.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.827+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f17d8007550 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:56:36.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.827+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f17d8007550 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:56:36.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.827+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:56:36.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.827+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:36.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.827+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:36.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.827+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:36.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.827+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17d8047ae0 2024-09-10T03:56:36.829 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.827+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:56:36.829 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:36.564+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:36.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:36.564+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:36.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:36.564+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:36.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:36.565+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:36.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.827+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:56:36.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.827+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:56:36.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.828+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:36.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.828+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80490b0 con 0x7f17d8047ae0 2024-09-10T03:56:36.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.828+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f17d801a350 tx=0x7f17e80358c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:56:36.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.829+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028040 con 0x7f17d8047ae0 2024-09-10T03:56:36.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.829+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0034e80 con 0x7f17d8047ae0 2024-09-10T03:56:36.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:36.830+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8047ae0 2024-09-10T03:56:37.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:56:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:56:36.827+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 7308 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:56:37.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:37.310+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:37.310+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:37.310+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:37.311+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:37.181+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:37.181+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:37.181+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:37.182+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:37.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:37.983+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:56:37.984 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:37.983+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:56:37.984 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:37.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:37.985 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:37.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:37.985 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:37.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:37.985 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:37.585+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:38.278 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:38.195+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:38.279 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:38.195+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:38.279 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:38.195+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:38.279 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:38.195+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:38.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:38.277+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:38.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:38.277+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:38.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:38.277+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:38.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:38.278+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:38.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:38.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:38.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:38.606+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:39.576 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:39.250+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:39.576 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:39.250+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:39.576 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:39.250+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:39.576 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:39.251+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:39.577 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:39.157+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:39.577 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:39.157+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:39.577 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:39.157+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:39.577 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:39.157+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:39.575+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:39.575+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:39.575+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:39.575+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:40.579 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:40.221+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:40.579 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:40.221+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:40.579 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:40.221+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:40.579 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:40.222+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:40.579 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:40.179+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:40.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:40.179+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:40.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:40.179+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:40.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:40.180+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:40.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:40.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:40.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:40.578+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:41.561 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:41.184+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:41.561 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:41.184+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:41.561 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:41.184+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:41.561 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:41.185+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:41.561 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:41.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:41.562 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:41.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:41.562 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:41.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:41.562 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:41.223+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:41.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.828+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:56:41.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.828+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:56:41.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.828+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f17d801a350 tx=0x7f17e80358c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:56:41.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.828+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f17d801a350 tx=0x7f17e80358c0 comp rx=0 tx=0).stop 2024-09-10T03:56:41.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.828+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:56:41.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.828+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:41.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.828+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:41.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.828+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:41.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.828+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17d805a5c0 2024-09-10T03:56:41.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.828+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:56:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:41.559+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:41.559+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:41.559+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:41.560+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:41.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.829+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:56:41.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.829+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:56:41.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.829+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:41.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.829+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80490b0 con 0x7f17d805a5c0 2024-09-10T03:56:41.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.830+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f17e8014fa0 tx=0x7f17e0050fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:56:41.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.830+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e801dd20 con 0x7f17d805a5c0 2024-09-10T03:56:41.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.831+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d805a5c0 2024-09-10T03:56:41.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:41.833+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c050 con 0x7f17d805a5c0 2024-09-10T03:56:42.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:56:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:56:41.828+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 7371 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:56:42.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:42.231+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:42.231+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:42.231+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:42.232+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:42.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:42.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:42.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:42.213+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:43.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:42.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:42.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:42.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:42.597+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:43.569 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:43.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:43.569 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:43.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:43.569 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:43.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:43.569 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:43.189+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:43.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:43.168+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:43.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:43.168+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:43.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:43.168+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:43.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:43.169+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:43.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:43.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:43.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:43.583+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:44.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:44.214+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:44.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:44.214+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:44.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:44.214+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:44.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:44.215+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:44.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:44.175+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:44.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:44.175+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:44.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:44.176+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:44.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:44.176+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:45.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:44.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:45.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:44.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:45.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:44.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:45.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:44.631+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:45.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:45.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:45.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:45.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:45.177+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:45.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:45.222+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:45.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:45.222+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:45.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:45.222+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:45.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:45.223+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:46.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:45.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:46.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:45.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:45.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:45.631+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:46.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:46.130+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:46.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:46.130+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:46.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:46.130+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:46.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:46.131+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:46.254+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:46.254+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:46.254+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:46.255+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:46.830 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.828+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:56:46.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.828+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:56:46.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.828+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f17e8014fa0 tx=0x7f17e0050fc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:56:46.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.828+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f17e8014fa0 tx=0x7f17e0050fc0 comp rx=0 tx=0).stop 2024-09-10T03:56:46.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.828+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:56:46.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.828+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:46.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.828+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:46.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.828+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:46.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.828+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17d8047ae0 2024-09-10T03:56:46.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.828+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:56:46.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.828+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:56:46.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.829+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:56:46.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.829+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:46.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.829+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80490b0 con 0x7f17d8047ae0 2024-09-10T03:56:46.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.830+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f17d804b250 tx=0x7f17e800f0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:56:46.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.830+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028040 con 0x7f17d8047ae0 2024-09-10T03:56:46.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.833+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e000dec0 con 0x7f17d8047ae0 2024-09-10T03:56:46.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:46.834+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d8047ae0 2024-09-10T03:56:47.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:56:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:56:46.828+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 7434 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:56:47.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:46.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:46.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:46.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:46.666+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:47.479 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:47.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:47.479 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:47.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:47.479 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:47.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:47.479 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:47.152+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:47.480 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:47.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:47.480 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:47.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:47.480 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:47.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:47.480 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:47.300+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:47.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:47.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:47.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:47.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:47.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:47.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:47.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:47.632+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:48.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:48.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:48.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:48.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:48.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:48.308+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:48.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:48.141+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:48.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:48.141+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:48.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:48.141+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:48.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:48.142+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:49.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:48.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:49.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:48.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:49.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:48.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:48.637+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:49.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:49.260+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:49.260+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:49.260+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:49.261+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:49.095+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:49.095+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:49.095+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:49.096+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:50.068 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:49.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:50.068 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:49.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:50.068 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:49.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:50.068 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:49.609+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:50.297+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:50.297+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:50.297+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:50.297+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:50.066+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:50.066+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:50.066+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:50.066+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:51.070 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:50.619+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:51.070 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:50.619+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:51.070 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:50.619+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:51.070 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:50.619+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:51.306+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:51.306+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:51.306+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:51.307+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:51.068+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:51.068+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:51.068+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:51.068+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:51.831 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.829+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:56:51.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.829+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:56:51.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.829+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f17d804b250 tx=0x7f17e800f0b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:56:51.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.829+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f17d804b250 tx=0x7f17e800f0b0 comp rx=0 tx=0).stop 2024-09-10T03:56:51.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.829+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:56:51.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.829+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:51.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.829+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:51.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.829+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:51.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.829+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17d805a5c0 2024-09-10T03:56:51.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.829+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:56:51.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.830+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:56:51.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.830+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:56:51.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.830+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:51.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.830+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80490b0 con 0x7f17d805a5c0 2024-09-10T03:56:51.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.831+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f17e805bff0 tx=0x7f17e001a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:56:51.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.831+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055050 con 0x7f17d805a5c0 2024-09-10T03:56:51.834 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:56:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:56:51.829+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 7496 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:56:51.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:51.583+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:51.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:51.583+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:51.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:51.583+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:51.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:51.583+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:51.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.832+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d805a5c0 2024-09-10T03:56:51.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:51.832+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064070 con 0x7f17d805a5c0 2024-09-10T03:56:52.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:52.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:52.298 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:52.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:52.298 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:52.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:52.298 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:52.035+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:52.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:52.296+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:52.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:52.296+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:52.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:52.296+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:52.297+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:53.018 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:52.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:53.019 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:52.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:53.019 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:52.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:53.019 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:52.599+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:53.284 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:53.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:53.285 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:53.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:53.285 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:53.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:53.285 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:53.018+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:53.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:53.407+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:56:53.408 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:53.407+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:56:53.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:53.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:53.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:53.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:53.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:53.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:53.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:53.284+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:53.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:53.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:53.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:53.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:53.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:53.612+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:54.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:54.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:54.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:54.061+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:54.292+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:54.292+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:54.292+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:54.293+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:55.026 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:54.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:55.026 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:54.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:55.026 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:54.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:55.026 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:54.646+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:55.291 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:55.025+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:55.291 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:55.025+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:55.291 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:55.025+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:55.292 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:55.025+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:55.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:55.290+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:55.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:55.290+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:55.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:55.290+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:55.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:55.291+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:56.059 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:55.683+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:56.060 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:55.683+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:56.060 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:55.683+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:56.060 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:55.684+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:56.058+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:56.058+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:56.058+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:56.058+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:56.257+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:56.257+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:56.257+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:56.258+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:56.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:56.706+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:56.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:56.706+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:56.706+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:56.706+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:56.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.831+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:56:56.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.831+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:56:56.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.831+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f17e805bff0 tx=0x7f17e001a8c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:56:56.832 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.831+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f17e805bff0 tx=0x7f17e001a8c0 comp rx=0 tx=0).stop 2024-09-10T03:56:56.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.831+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:56:56.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.831+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:56.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.831+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:56.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.831+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:56:56.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.831+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17d8047ae0 2024-09-10T03:56:56.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.831+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:56:56.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.831+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:56:56.833 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.832+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:56:56.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.832+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:56:56.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.832+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048ee0 con 0x7f17d8047ae0 2024-09-10T03:56:56.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.833+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009f50 tx=0x7f17e8009d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:56:56.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.835+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e803f400 con 0x7f17d8047ae0 2024-09-10T03:56:56.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.835+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d8047ae0 2024-09-10T03:56:56.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:56:56.835+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064040 con 0x7f17d8047ae0 2024-09-10T03:56:57.233 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:57.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:57.233 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:57.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:57.233 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:57.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:57.233 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:57.019+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:57.233 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:56:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:56:56.831+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 7557 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:56:57.234 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:57.231+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:57.234 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:57.231+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:57.234 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:57.231+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:57.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:57.232+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:58.037 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:57.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:58.037 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:57.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:58.037 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:57.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:58.037 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:57.749+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:58.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:58.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:58.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:58.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:58.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:58.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:58.036+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:58.282+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:58.282+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:58.282+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:58.283+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:59.004 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:58.729+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:59.004 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:58.729+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:59.004 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:58.729+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:56:59.004 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:58.730+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:56:59.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:59.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:59.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:59.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:59.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:59.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:56:59.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:56:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:56:59.003+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:56:59.714 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:59.332+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:59.715 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:59.332+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:59.715 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:59.332+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:56:59.715 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:56:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:56:59.333+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:00.014 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:59.713+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:00.015 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:59.713+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:00.015 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:59.713+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:00.015 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:56:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:56:59.714+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:00.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:00.013+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:00.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:00.013+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:00.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:00.013+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:00.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:00.014+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:00.694 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:00.348+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:00.695 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:00.348+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:00.695 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:00.348+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:00.695 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:00.348+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:01.006 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:00.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:01.006 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:00.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:01.006 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:00.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:01.006 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:00.694+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:01.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:01.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:01.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:01.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:01.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:01.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:01.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:01.006+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:01.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:01.375+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:01.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:01.375+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:01.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:01.375+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:01.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:01.376+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:01.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.832+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:57:01.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.832+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:57:01.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.832+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009f50 tx=0x7f17e8009d20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:57:01.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.832+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009f50 tx=0x7f17e8009d20 comp rx=0 tx=0).stop 2024-09-10T03:57:01.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.832+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:57:01.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.832+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:57:01.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.832+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:57:01.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.832+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:57:01.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.832+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17d805a5c0 2024-09-10T03:57:01.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.832+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:57:01.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.832+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:57:01.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.833+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:57:01.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.833+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:57:01.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.833+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80490b0 con 0x7f17d805a5c0 2024-09-10T03:57:01.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.833+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f17e0055fe0 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:57:01.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.834+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c040 con 0x7f17d805a5c0 2024-09-10T03:57:01.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.834+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e003c1e0 con 0x7f17d805a5c0 2024-09-10T03:57:01.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:01.835+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d805a5c0 2024-09-10T03:57:02.006 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:57:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:57:01.833+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 7619 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:57:02.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:01.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:02.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:01.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:02.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:01.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:02.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:01.664+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:02.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:02.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:02.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:02.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:02.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:02.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:02.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:02.005+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:02.625 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:02.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:02.625 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:02.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:02.625 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:02.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:02.625 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:02.372+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:02.957 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:02.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:02.957 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:02.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:02.957 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:02.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:02.957 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:02.626+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:03.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:02.956+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:03.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:02.956+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:03.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:02.956+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:03.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:02.957+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:03.597 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:03.330+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:03.597 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:03.330+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:03.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:03.330+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:03.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:03.331+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:03.985 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:03.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:03.985 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:03.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:03.985 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:03.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:03.985 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:03.597+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:04.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:03.984+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:04.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:03.984+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:03.984+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:03.984+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:04.613 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:04.354+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:04.613 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:04.354+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:04.613 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:04.354+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:04.613 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:04.354+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:04.998 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:04.611+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:04.998 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:04.611+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:04.998 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:04.611+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:04.998 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:04.612+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:05.312 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:04.997+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:05.312 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:04.997+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:05.312 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:04.997+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:05.312 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:04.997+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:05.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:05.382+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:05.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:05.382+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:05.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:05.382+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:05.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:05.383+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:05.978 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:05.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:05.978 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:05.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:05.978 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:05.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:05.978 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:05.599+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:06.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:05.976+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:06.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:05.976+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:05.976+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:05.977+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:06.621 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:06.366+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:06.621 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:06.366+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:06.621 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:06.366+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:06.621 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:06.366+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:06.834 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.833+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:57:06.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.833+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:57:06.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.833+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f17e0055fe0 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:57:06.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.833+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f17e0055fe0 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:57:06.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.833+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:57:06.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.833+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:57:06.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.833+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:57:06.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.833+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:57:06.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.833+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048220 con 0x7f17d8047ae0 2024-09-10T03:57:06.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.833+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:57:06.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.834+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:57:06.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.834+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:57:06.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.834+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:57:06.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.834+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8049280 con 0x7f17d8047ae0 2024-09-10T03:57:06.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.834+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f17e802bba0 tx=0x7f17e0021ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:57:06.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.836+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e0028070 con 0x7f17d8047ae0 2024-09-10T03:57:06.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.836+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e003c1e0 con 0x7f17d8047ae0 2024-09-10T03:57:06.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:06.837+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063040 con 0x7f17d8047ae0 2024-09-10T03:57:06.969 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:57:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:57:06.833+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 7682 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:57:06.969 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:06.619+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:06.970 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:06.619+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:06.970 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:06.619+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:06.970 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:06.620+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:07.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:06.968+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:07.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:06.968+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:07.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:06.968+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:06.969+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:07.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:07.406+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:07.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:07.406+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:07.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:07.406+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:07.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:07.407+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:07.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:07.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:07.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:07.592+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:08.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:07.962+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:08.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:07.962+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:07.962+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:07.962+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:08.634 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:08.381+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:08.634 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:08.381+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:08.635 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:08.381+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:08.635 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:08.382+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:08.768 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:08.767+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:57:08.769 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:08.767+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:57:08.962 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:08.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:08.962 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:08.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:08.962 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:08.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:08.962 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:08.634+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:09.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:08.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:09.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:08.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:09.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:08.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:09.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:08.961+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:09.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:09.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:09.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:09.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:09.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:09.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:09.629+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:09.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:09.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:09.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:09.427+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:10.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:09.957+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:10.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:09.957+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:09.957+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:09.958+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:10.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:10.473+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:10.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:10.473+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:10.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:10.473+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:10.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:10.474+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:10.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:10.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:10.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:10.586+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:11.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:10.910+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:11.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:10.910+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:11.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:10.910+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:11.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:10.911+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:11.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:11.435+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:11.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:11.435+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:11.435+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:11.436+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:11.623+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:11.623+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:11.623+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:11.624+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:11.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.834+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:57:11.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.834+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:57:11.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.834+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f17e802bba0 tx=0x7f17e0021ef0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:57:11.835 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.834+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f17e802bba0 tx=0x7f17e0021ef0 comp rx=0 tx=0).stop 2024-09-10T03:57:11.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.834+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:57:11.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.834+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:57:11.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.834+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:57:11.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.834+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:57:11.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.834+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048220 con 0x7f17d805a5c0 2024-09-10T03:57:11.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.834+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:57:11.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.834+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:57:11.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.835+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:57:11.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.835+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:57:11.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.835+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80490b0 con 0x7f17d805a5c0 2024-09-10T03:57:11.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.835+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f17e803f7b0 tx=0x7f17e0047370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:57:11.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.836+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8003a30 con 0x7f17d805a5c0 2024-09-10T03:57:11.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.836+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d805a5c0 2024-09-10T03:57:11.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:11.837+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c050 con 0x7f17d805a5c0 2024-09-10T03:57:12.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:57:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:57:11.834+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 7745 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:57:12.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:11.929+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:11.929+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:11.929+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:11.929+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:12.646 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:12.387+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:12.646 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:12.387+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:12.646 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:12.387+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:12.646 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:12.388+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:12.958 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:12.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:12.958 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:12.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:12.958 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:12.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:12.958 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:12.645+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:13.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:12.957+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:13.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:12.957+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:13.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:12.957+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:12.957+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:13.681 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:13.406+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:13.681 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:13.406+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:13.681 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:13.406+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:13.681 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:13.407+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:13.950 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:13.680+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:13.950 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:13.680+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:13.950 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:13.680+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:13.950 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:13.680+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:13.950 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:13.948+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:13.950 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:13.948+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:13.950 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:13.948+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:14.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:13.949+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:14.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:14.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:14.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:14.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:14.416+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:14.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:14.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:14.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:14.655+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:14.950+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:14.950+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:14.950+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:14.951+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:15.706 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:15.442+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:15.706 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:15.442+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:15.706 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:15.442+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:15.706 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:15.442+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:15.982 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:15.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:15.982 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:15.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:15.982 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:15.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:15.982 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:15.705+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:16.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:15.980+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:16.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:15.980+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:15.980+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:15.981+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:16.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:16.468+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:16.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:16.468+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:16.468+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:16.469+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:16.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:16.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:16.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:16.709+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.835+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:57:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.835+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:57:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.835+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f17e803f7b0 tx=0x7f17e0047370 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:57:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.835+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f17e803f7b0 tx=0x7f17e0047370 comp rx=0 tx=0).stop 2024-09-10T03:57:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.835+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:57:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.835+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:57:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.835+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:57:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.835+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:57:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.835+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048220 con 0x7f17d8047ae0 2024-09-10T03:57:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.835+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:57:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.835+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:57:16.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.836+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:57:16.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.836+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:57:16.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.836+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80490b0 con 0x7f17d8047ae0 2024-09-10T03:57:16.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.836+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f17e802b9d0 tx=0x7f17e800fd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:57:16.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.836+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e802ba00 con 0x7f17d8047ae0 2024-09-10T03:57:16.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.837+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b070 con 0x7f17d8047ae0 2024-09-10T03:57:16.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:16.838+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e801c050 con 0x7f17d8047ae0 2024-09-10T03:57:17.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:57:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:57:16.835+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 7808 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:57:17.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:16.939+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:17.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:16.939+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:16.939+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:16.939+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:17.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:17.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:17.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:17.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:17.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:17.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:17.667+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:17.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:17.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:17.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:17.511+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:17.918+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:17.918+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:17.918+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:17.919+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:18.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:18.494+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:18.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:18.494+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:18.494+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:18.495+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:18.619+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:18.619+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:18.619+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:18.620+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:19.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:18.957+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:19.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:18.957+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:18.957+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:18.958+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:19.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:19.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:19.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:19.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:19.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:19.507+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:19.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:19.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:19.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:19.633+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:20.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:19.991+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:20.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:19.991+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:20.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:19.991+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:20.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:19.991+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:20.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:20.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:20.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:20.671+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:20.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:20.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:20.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:20.524+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:21.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:21.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:21.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:21.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:21.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:21.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:21.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:21.040+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:21.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:21.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:21.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:21.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:21.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:21.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:21.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:21.503+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:21.667+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:21.667+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:21.667+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:21.668+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:21.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.836+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T03:57:21.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.836+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:57:21.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.836+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f17e802b9d0 tx=0x7f17e800fd00 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:57:21.837 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.836+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f17e802b9d0 tx=0x7f17e800fd00 comp rx=0 tx=0).stop 2024-09-10T03:57:21.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.836+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:57:21.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.836+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:57:21.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.836+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:57:21.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.836+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:57:21.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.836+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17d805a5c0 2024-09-10T03:57:21.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.836+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:57:21.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.836+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:57:21.838 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.837+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:57:21.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.837+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:57:21.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.837+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80490b0 con 0x7f17d805a5c0 2024-09-10T03:57:21.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.838+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f17e8040d40 tx=0x7f17e800f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:57:21.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.838+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e8055050 con 0x7f17d805a5c0 2024-09-10T03:57:21.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.839+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e803b040 con 0x7f17d805a5c0 2024-09-10T03:57:21.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:21.840+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064040 con 0x7f17d805a5c0 2024-09-10T03:57:22.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:57:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:57:21.837+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 7871 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:57:22.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:22.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:22.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:22.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:22.020+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:22.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:22.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:22.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:22.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:22.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:22.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:22.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:22.686+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:22.481+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:22.481+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:22.481+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:22.481+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:23.237 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:22.984+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:23.238 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:22.984+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:23.238 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:22.984+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:23.238 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:22.985+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:23.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:23.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:23.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:23.697+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:23.485+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:23.485+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:23.485+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:23.486+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:24.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:24.126+0000 7f17edd9b640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 msgr2=0x7f17d8041e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.169:6828/1204620466 2024-09-10T03:57:24.128 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:24.126+0000 7f17edd9b640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:6828/1204620466,v1:172.21.15.169:6829/1204620466] conn(0x7f17d803f9c0 0x7f17d8041e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-10T03:57:24.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:24.011+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:24.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:24.011+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:24.011+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:24.011+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:24.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:24.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:24.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:24.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:24.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:24.738+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:24.496+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:24.496+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:24.496+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:24.497+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:25.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:25.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:25.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:25.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:25.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:25.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:25.019+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:25.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:25.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:25.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:25.759+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:25.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:25.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:25.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:25.534+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:26.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:26.045+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:26.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:26.045+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:26.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:26.045+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:26.046+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:26.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:26.763+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:26.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:26.763+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:26.763+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:26.763+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:26.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:26.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:26.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:26.558+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:26.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.837+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:57:26.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.837+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:57:26.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.837+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f17e8040d40 tx=0x7f17e800f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:57:26.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.837+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f17e8040d40 tx=0x7f17e800f040 comp rx=0 tx=0).stop 2024-09-10T03:57:26.839 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.837+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 msgr2=0x7f17d804bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:57:26.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.837+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:57:26.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.837+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:57:26.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.837+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:57:26.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.837+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17d8047ae0 2024-09-10T03:57:26.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.837+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:57:26.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.837+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:57:26.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.838+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:57:26.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.838+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:57:26.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.838+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d80490b0 con 0x7f17d8047ae0 2024-09-10T03:57:26.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.839+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f17e800d880 tx=0x7f17e8037040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:57:26.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.841+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e804ea20 con 0x7f17d8047ae0 2024-09-10T03:57:26.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.841+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e805be40 con 0x7f17d8047ae0 2024-09-10T03:57:26.844 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:26.841+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e8064040 con 0x7f17d8047ae0 2024-09-10T03:57:27.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:57:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:57:26.837+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 7934 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:57:27.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:27.084+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:27.084+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:27.084+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:27.085+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:27.764+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:27.764+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:27.764+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:27.765+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:27.565+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:27.565+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:27.565+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:27.566+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:28.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:28.046+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:28.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:28.046+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:28.046+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:28.047+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:29.052 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:28.599+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:29.052 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:28.599+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:29.052 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:28.599+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:29.053 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:28.600+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:29.053 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:28.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:29.053 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:28.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:29.053 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:28.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:29.053 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:28.794+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:29.053 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:29.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:29.053 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:29.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:29.054 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:29.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:29.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:29.050+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:29.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:29.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:29.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:29.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:29.558+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:29.765+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:29.765+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:29.765+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:29.766+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:30.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:30.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:30.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:30.081+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:31.038 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:30.781+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:31.038 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:30.781+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:31.038 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:30.781+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:31.038 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:30.782+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:31.039 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:30.577+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:31.039 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:30.577+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:31.039 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:30.577+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:31.039 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:30.577+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:31.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:31.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:31.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:31.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:31.038+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:31.776+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:31.776+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:31.776+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:31.776+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:31.565+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:31.565+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:31.565+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:31.566+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:31.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.838+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T03:57:31.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.838+0000 7f17eed9d640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T03:57:31.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.838+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f17e800d880 tx=0x7f17e8037040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T03:57:31.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.838+0000 7f17eed9d640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f17e800d880 tx=0x7f17e8037040 comp rx=0 tx=0).stop 2024-09-10T03:57:31.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.838+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 msgr2=0x7f17d80486a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T03:57:31.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.839+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d8047ae0 0x7f17d80486a0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:57:31.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.839+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:57:31.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.839+0000 7f17df7fe640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T03:57:31.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.839+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048170 con 0x7f17d805a5c0 2024-09-10T03:57:31.840 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.839+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8048be0 con 0x7f17f0101a40 2024-09-10T03:57:31.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.839+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T03:57:31.841 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.840+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 msgr2=0x7f17d8050580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T03:57:31.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.840+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f17f0101a40 0x7f17d8050580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T03:57:31.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.840+0000 7f17ee59c640 1 -- 172.21.15.112:0/2221466548 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) -- 0x7f17d8048ee0 con 0x7f17d805a5c0 2024-09-10T03:57:31.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.840+0000 7f17ee59c640 1 --2- 172.21.15.112:0/2221466548 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17d805a5c0 0x7f17d804bdd0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f17e0022e20 tx=0x7f17e8040690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T03:57:31.842 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.841+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f17e003c040 con 0x7f17d805a5c0 2024-09-10T03:57:31.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.842+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f17e0014340 con 0x7f17d805a5c0 2024-09-10T03:57:31.843 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T03:57:31.842+0000 7f17df7fe640 1 -- 172.21.15.112:0/2221466548 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mgrmap(e 23) ==== 51997+0+0 (secure 0 0 0) 0x7f17e0063070 con 0x7f17d805a5c0 2024-09-10T03:57:32.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:57:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:57:31.839+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 7997 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:57:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:31.998+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:31.998+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:31.998+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:32.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:31.999+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:33.045 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:32.607+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:33.046 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:32.607+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:33.046 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:32.607+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:33.046 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:32.608+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:33.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:32.826+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:33.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:32.826+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:33.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:32.826+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:33.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:32.826+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:33.044+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:33.044+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:33.044+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:33.045+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:33.824 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:33.574+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:33.824 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:33.574+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:33.824 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:33.574+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:33.824 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:33.574+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:34.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:33.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:34.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:33.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:34.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:33.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:33.824+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:34.050+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:34.050+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:34.050+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:34.050+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:34.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:34.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:34.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:34.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:34.563+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:35.087 INFO:teuthology.orchestra.run.smithi112.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-09-10T03:57:35.087 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:35.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:35.087 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:35.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:35.087 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:35.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:35.087 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:35.028+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:35.088 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:34.829+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:35.088 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:34.829+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:35.088 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:34.829+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:35.088 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:34.830+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:35.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:35.610+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:35.610+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:35.610+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:35.610+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:35.910 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -e sha1=3a5e3f16720f432aea965c40a8f6eb480095ec6c -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-09-10T03:57:36.182 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T03:57:36.306 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:36.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:36.306 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:36.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:36.306 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:36.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:36.307 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:36.065+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:36.307 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:35.838+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:36.307 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:35.838+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:36.307 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:35.838+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:36.307 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:35.839+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:36.842 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:36.812+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:36.842 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:36.812+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:36.842 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:36.812+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:36.842 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:36.812+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:36.842 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:36.647+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:36.842 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:36.648+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:36.842 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:36.648+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:36.842 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:36.648+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:37.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:57:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:57:36.840+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 8060 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:57:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:37.026+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:37.026+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:37.026+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:37.027+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:38.075 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:37.788+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:38.075 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:37.788+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:38.075 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:37.788+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:38.075 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:37.788+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:38.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:37.661+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:38.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:37.661+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:38.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:37.661+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:38.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:37.662+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:38.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:38.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:38.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:38.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:38.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:38.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:38.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:38.075+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:38.788 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:38.665+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:38.788 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:38.665+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:38.788 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:38.665+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:38.788 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:38.666+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:39.067 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:38.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:39.067 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:38.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:39.067 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:38.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:39.067 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:38.787+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:39.066+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:39.066+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:39.066+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:39.066+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:40.025 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:39.697+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:40.025 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:39.697+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:40.025 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:39.697+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:40.025 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:39.698+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:40.025 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:39.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:40.025 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:39.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:40.025 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:39.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:40.025 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:39.748+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:40.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:40.023+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:40.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:40.023+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:40.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:40.023+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:40.024+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:41.044 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:40.648+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:41.045 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:40.648+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:41.045 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:40.648+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:41.045 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:40.649+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:41.045 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:40.797+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:41.045 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:40.797+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:41.045 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:40.797+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:41.045 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:40.798+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:41.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:41.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:41.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:41.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:41.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:41.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:41.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:41.044+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:42.061 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:41.818+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:42.061 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:41.818+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:42.061 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:41.818+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:42.061 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:41.819+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:42.062 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:41.637+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:42.062 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:41.637+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:42.062 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:41.637+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:42.062 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:41.638+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:42.062 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:57:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:57:41.841+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 8123 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:57:42.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:42.060+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:42.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:42.060+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:42.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:42.060+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:42.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:42.060+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:43.038 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:42.642+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:43.038 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:42.642+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:43.038 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:42.642+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:43.039 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:42.643+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:43.039 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:42.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:43.039 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:42.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:43.039 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:42.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:43.039 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:42.822+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:43.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:43.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:43.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:43.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:43.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:43.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:43.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:43.037+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:43.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:43.651+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:43.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:43.651+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:43.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:43.651+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:43.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:43.653+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:43.804+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:43.804+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:43.804+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:43.805+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:44.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:44.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:44.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:44.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:44.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:44.017+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:45.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:44.650+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:45.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:44.650+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:45.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:44.650+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:45.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:44.651+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:45.065 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:44.765+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:45.065 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:44.765+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:45.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:44.765+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:45.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:44.766+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:45.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:45.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:45.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:45.064+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:46.041 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:45.788+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:46.041 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:45.788+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:46.041 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:45.788+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:46.041 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:45.789+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:46.041 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:45.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:46.041 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:45.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:46.042 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:45.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:46.042 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:45.616+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:46.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:46.039+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:46.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:46.039+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:46.039+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:46.040+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:47.029 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:46.745+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:47.029 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:46.745+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:47.029 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:46.745+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:47.029 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:46.746+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:47.029 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:46.650+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:47.030 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:46.650+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:47.030 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:46.651+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:47.030 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:46.651+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:47.030 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:57:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:57:46.842+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 8187 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:57:47.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:47.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:47.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:47.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:47.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:47.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:47.028+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:47.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:47.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:47.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:47.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:47.631+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:47.631+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:47.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:47.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:47.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:47.773+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:48.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:48.046+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:48.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:48.046+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:48.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:48.046+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:48.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:48.046+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:48.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:48.667+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:48.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:48.667+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:48.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:48.667+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:48.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:48.668+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:49.074 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:48.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:49.074 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:48.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:49.074 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:48.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:49.074 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:48.732+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:49.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:49.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:49.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:49.073+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:50.025 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:49.653+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:50.026 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:49.653+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:50.026 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:49.653+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:50.026 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:49.654+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:50.026 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:49.715+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:50.026 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:49.715+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:50.026 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:49.715+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:50.026 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:49.715+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:50.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:50.024+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:50.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:50.024+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:50.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:50.024+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:50.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:50.025+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:51.074 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:50.683+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:51.074 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:50.683+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:51.074 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:50.683+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:51.074 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:50.684+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:51.074 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:50.736+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:51.074 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:50.736+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:51.074 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:50.736+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:51.075 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:50.737+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:51.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:51.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:51.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:51.073+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:52.050 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:51.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:52.050 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:51.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:52.050 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:51.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:52.050 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:51.688+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:52.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:51.681+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:52.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:51.681+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:52.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:51.681+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:52.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:51.682+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:52.051 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:57:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:57:51.843+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 8249 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:57:52.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:52.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:52.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:52.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:52.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:52.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:52.050+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:52.714+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:52.714+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:52.714+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:52.715+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:53.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:52.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:53.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:52.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:53.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:52.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:53.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:52.688+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:53.573 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:53.100+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:53.573 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:53.100+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:53.573 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:53.100+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:53.573 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:53.100+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:53.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:53.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:53.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:53.738+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:53.764+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:53.764+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:53.764+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:53.765+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:54.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:54.107+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:54.107+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:54.107+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:54.108+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:54.913 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:54.740+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:54.913 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:54.740+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:54.913 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:54.740+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:54.913 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:54.741+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:54.913 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:54.731+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:54.913 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:54.731+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:54.914 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:54.731+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:54.914 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:54.732+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:55.079+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:55.079+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:55.079+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:55.080+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:56.074 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:55.755+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:56.074 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:55.755+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:56.074 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:55.755+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:56.074 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:55.756+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:56.074 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:55.780+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:56.074 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:55.780+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:56.074 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:55.780+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:56.074 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:55.780+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:56.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:56.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:56.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:56.073+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:57.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:56.723+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:57.035 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:56.723+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:57.035 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:56.723+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:57.035 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:56.724+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:57.035 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:56.805+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:57.035 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:56.805+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:57.035 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:56.805+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:57.035 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:56.805+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:57.036 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:57:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:57:56.844+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 8312 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:57:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:57.033+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:57.033+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:57.033+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:57.034+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:57.985 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:57.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:57.985 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:57.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:57.985 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:57.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:57.985 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:57.754+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:57.985 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:57.842+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:57.986 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:57.842+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:57.986 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:57.842+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:57.986 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:57.843+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:58.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:58.075+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:58.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:58.075+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:58.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:58.075+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:58.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:58.075+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:59.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:58.827+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:58.827+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:58.827+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:57:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:58.828+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:57:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:58.707+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:58.707+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:58.707+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:57:59.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:58.708+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:57:59.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:59.092+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:59.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:59.092+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:59.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:59.092+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:57:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:57:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:57:59.093+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:00.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:59.756+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:00.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:59.756+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:00.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:59.756+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:00.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:57:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:57:59.757+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:59.830+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:59.830+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:59.830+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:57:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:57:59.830+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:00.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:00.108+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:00.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:00.108+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:00.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:00.108+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:00.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:00.109+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:01.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:00.854+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:01.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:00.854+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:01.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:00.854+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:01.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:00.855+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:01.058 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:00.747+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:01.058 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:00.747+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:01.058 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:00.747+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:01.058 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:00.747+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:01.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:01.081+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:01.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:01.081+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:01.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:01.081+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:01.081+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:02.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:01.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:01.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:01.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:01.824+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:01.745+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:01.745+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:01.745+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:01.746+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:02.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:58:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:58:01.844+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 8375 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:58:02.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:02.099+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:02.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:02.099+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:02.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:02.099+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:02.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:02.100+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:03.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:02.815+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:03.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:02.815+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:03.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:02.815+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:02.816+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:02.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:02.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:02.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:02.766+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:03.573 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:03.093+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:03.573 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:03.093+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:03.573 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:03.093+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:03.573 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:03.094+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:03.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:03.779+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:03.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:03.779+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:03.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:03.779+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:03.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:03.779+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:04.128 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:03.844+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:04.129 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:03.844+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:04.129 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:03.844+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:04.129 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:03.845+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:04.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:04.143+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:04.143+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:04.143+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:04.143+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:04.891 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:04.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:04.891 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:04.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:04.891 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:04.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:04.891 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:04.783+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:05.179 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:04.890+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:05.179 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:04.890+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:05.179 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:04.890+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:05.179 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:04.890+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:05.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:05.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:05.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:05.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:05.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:05.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:05.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:05.177+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:06.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:05.805+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:05.805+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:05.805+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:05.805+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:05.846+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:05.846+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:05.846+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:05.847+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:06.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:06.167+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:06.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:06.167+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:06.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:06.167+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:06.168+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:07.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:06.815+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:07.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:06.815+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:07.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:06.815+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:07.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:06.816+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:06.853+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:06.853+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:06.853+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:06.854+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:07.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:58:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:58:06.846+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 8438 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:58:07.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:07.128+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:07.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:07.128+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:07.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:07.128+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:07.128+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:08.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:07.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:07.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:07.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:07.845+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:07.824+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:07.824+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:07.824+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:07.825+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:08.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:08.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:08.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:08.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:08.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:08.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:08.152+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:09.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:08.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:08.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:08.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:08.818+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:09.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:08.833+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:09.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:08.833+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:09.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:08.833+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:09.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:08.834+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:09.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:09.107+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:09.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:09.107+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:09.107+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:09.108+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:10.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:09.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:10.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:09.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:10.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:09.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:10.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:09.876+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:09.770+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:09.770+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:09.770+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:09.770+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:10.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:10.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:10.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:10.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:10.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:10.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:10.099+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:11.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:10.740+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:11.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:10.740+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:10.740+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:10.741+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:11.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:10.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:11.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:10.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:11.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:10.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:11.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:10.908+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:11.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:11.114+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:11.114+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:11.114+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:11.114+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:12.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:11.701+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:11.701+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:11.701+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:11.702+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:11.944+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:11.944+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:11.944+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:11.945+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:12.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:58:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:58:11.847+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 8501 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:58:12.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:12.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:12.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:12.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:12.090+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:12.943 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:12.680+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:12.943 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:12.680+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:12.943 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:12.680+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:12.943 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:12.680+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:13.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:13.071+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:13.071+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:13.071+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:13.072+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:12.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:12.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:12.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:12.942+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:13.717+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:13.717+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:13.717+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:13.718+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:14.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:14.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:14.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:14.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:14.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:14.080+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:14.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:13.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:14.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:13.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:14.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:13.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:14.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:13.918+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:14.900 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:14.898+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:14.901 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:14.898+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:14.901 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:14.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:14.901 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:14.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:14.901 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:14.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:14.902 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:14.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:14.902 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:14.749+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:14.899+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:15.090+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:15.090+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:15.090+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:15.090+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:16.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:15.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:16.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:15.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:16.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:15.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:16.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:15.921+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:16.076 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:15.727+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:16.077 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:15.727+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:16.077 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:15.727+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:16.077 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:15.728+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:16.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:16.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:16.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:16.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:16.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:16.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:16.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:16.075+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:17.069 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:16.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:17.069 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:16.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:17.069 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:16.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:17.069 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:16.690+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:17.070 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:58:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:58:16.848+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 8564 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:58:17.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:16.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:17.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:16.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:17.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:16.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:17.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:16.872+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:17.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:17.067+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:17.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:17.067+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:17.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:17.067+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:17.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:17.068+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:18.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:17.864+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:18.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:17.864+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:18.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:17.864+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:18.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:17.865+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:18.079 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:17.723+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:18.079 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:17.723+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:18.079 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:17.723+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:18.079 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:17.724+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:18.076+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:18.076+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:18.076+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:18.077+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:19.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:18.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:19.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:18.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:18.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:18.882+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:18.705+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:18.705+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:18.705+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:18.705+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:19.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:19.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:19.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:19.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:19.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:19.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:19.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:19.098+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:19.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:19.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:19.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:19.846+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:20.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:19.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:20.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:19.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:20.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:19.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:20.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:19.709+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:20.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:20.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:20.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:20.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:20.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:20.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:20.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:20.090+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:21.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:20.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:21.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:20.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:21.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:20.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:20.875+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:20.725+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:20.725+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:20.725+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:20.726+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:21.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:21.109+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:21.109+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:21.109+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:21.110+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:22.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:58:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:58:21.849+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 8627 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:58:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:21.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:21.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:21.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:21.898+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:21.730+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:21.730+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:21.730+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:21.731+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:22.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:22.118+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:22.118+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:22.118+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:22.119+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:23.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:22.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:23.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:22.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:23.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:22.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:23.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:22.879+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:22.718+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:22.718+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:22.718+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:22.719+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:23.574 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:23.095+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:23.574 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:23.095+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:23.575 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:23.095+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:23.575 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:23.096+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:23.689+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:23.689+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:23.689+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:23.690+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:24.270 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:24.097+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:24.270 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:24.097+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:24.270 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:24.097+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:24.270 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:24.098+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:24.270 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:23.856+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:24.270 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:23.856+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:24.270 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:23.856+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:24.270 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:23.857+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:25.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:24.836+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:25.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:24.836+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:25.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:24.836+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:25.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:24.837+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:25.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:24.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:25.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:24.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:25.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:24.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:25.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:24.738+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:25.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:25.119+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:25.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:25.119+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:25.119+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:25.120+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:25.884 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:25.718+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:25.884 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:25.718+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:25.884 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:25.718+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:25.884 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:25.719+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:26.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:26.131+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:26.131+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:26.131+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:26.132+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:25.883+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:25.883+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:26.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:25.883+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:26.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:25.883+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:27.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:58:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:58:26.850+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 8690 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:58:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:26.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:26.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:26.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:26.871+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:26.755+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:26.755+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:26.755+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:26.756+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:27.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:27.172+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:27.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:27.172+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:27.172+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:27.173+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:28.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:27.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:28.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:27.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:28.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:27.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:28.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:27.744+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:27.888+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:27.888+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:27.888+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:27.889+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:28.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:28.160+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:28.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:28.160+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:28.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:28.160+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:28.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:28.161+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:29.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:28.746+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:29.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:28.746+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:28.746+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:28.746+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:28.848+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:28.848+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:28.848+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:28.849+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:29.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:29.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:29.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:29.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:29.157+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:30.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:29.885+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:30.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:29.885+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:29.885+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:29.886+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:29.742+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:29.743+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:29.743+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:29.743+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:30.156+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:30.156+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:30.156+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:30.157+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:31.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:30.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:31.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:30.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:30.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:30.882+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:30.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:30.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:30.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:30.712+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:31.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:31.133+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:31.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:31.133+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:31.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:31.133+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:31.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:31.133+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:32.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:58:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:58:31.851+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 8752 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:58:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:31.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:31.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:31.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:31.871+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:31.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:31.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:31.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:31.731+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:32.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:32.115+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:32.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:32.115+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:32.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:32.115+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:32.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:32.116+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:33.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:32.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:33.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:32.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:33.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:32.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:33.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:32.890+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:32.745+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:32.745+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:32.745+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:32.746+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:33.575 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:33.156+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:33.575 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:33.156+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:33.576 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:33.156+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:33.576 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:33.157+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:33.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:33.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:33.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:33.732+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:34.185 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:33.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:34.186 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:33.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:34.186 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:33.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:34.186 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:33.929+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:34.471 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:34.184+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:34.471 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:34.184+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:34.471 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:34.184+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:34.472 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:34.185+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:34.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:34.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:34.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:34.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:34.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:34.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:34.687+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:35.160 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:34.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:35.160 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:34.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:35.160 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:34.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:35.160 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:34.880+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:35.553 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:35.158+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:35.553 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:35.158+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:35.553 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:35.158+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:35.553 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:35.159+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:36.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:35.713+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:35.713+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:35.713+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:35.713+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:36.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:35.887+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:36.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:35.888+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:36.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:35.888+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:36.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:35.888+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:36.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:36.145+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:36.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:36.145+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:36.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:36.145+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:36.146+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:37.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:36.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:37.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:36.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:37.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:36.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:37.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:36.731+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:36.884+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:36.884+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:36.884+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:36.885+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:37.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:58:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:58:36.852+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 8814 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:58:37.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:37.119+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:37.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:37.119+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:37.119+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:37.120+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:38.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:37.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:38.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:37.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:38.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:37.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:37.923+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:37.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:37.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:37.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:37.754+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:38.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:38.169+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:38.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:38.169+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:38.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:38.169+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:38.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:38.169+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:38.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:38.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:38.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:38.964+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:38.791+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:38.791+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:38.791+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:39.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:38.792+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:39.552 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:39.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:39.552 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:39.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:39.552 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:39.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:39.552 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:39.123+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:39.752+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:39.752+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:39.752+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:39.752+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:39.997+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:39.997+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:39.997+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:39.998+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:40.086+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:40.086+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:40.086+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:40.086+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:41.043 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:40.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:41.043 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:40.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:41.043 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:40.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:41.043 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:40.745+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:41.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:41.054+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:41.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:41.054+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:41.054+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:41.055+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:41.041+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:41.041+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:41.041+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:41.042+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:42.010 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:41.740+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:42.010 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:41.740+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:42.010 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:41.740+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:42.010 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:41.740+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:42.010 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:58:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:58:41.852+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 8875 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:58:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:42.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:42.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:42.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:42.029+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:42.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:42.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:42.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:42.008+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:42.996 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:42.732+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:42.996 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:42.732+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:42.997 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:42.732+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:42.997 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:42.732+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:43.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:42.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:43.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:42.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:42.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:42.995+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:43.025+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:43.025+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:43.025+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:43.025+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:43.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:43.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:43.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:43.735+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:44.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:44.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:44.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:44.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:44.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:44.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:44.041+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:44.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:44.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:44.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:44.076+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:45.048 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:44.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:45.048 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:44.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:45.049 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:44.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:45.049 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:44.699+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:45.047+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:45.047+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:45.047+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:45.048+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:45.123+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:45.123+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:45.123+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:45.124+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:46.048 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:45.673+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:46.048 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:45.673+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:46.048 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:45.673+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:46.048 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:45.674+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:46.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:46.047+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:46.047+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:46.047+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:46.047+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:46.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:46.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:46.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:46.149+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:47.076 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:46.702+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:47.076 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:46.702+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:47.076 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:46.702+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:47.076 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:46.703+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:47.076 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:58:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:58:46.855+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 8936 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:58:47.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:47.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:47.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:47.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:47.075+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:47.177+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:47.177+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:47.177+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:47.177+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:47.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:47.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:47.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:47.688+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:48.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:48.079+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:48.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:48.079+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:48.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:48.079+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:48.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:48.080+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:48.144+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:48.144+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:48.144+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:48.145+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:49.048 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:48.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:49.048 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:48.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:49.049 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:48.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:49.049 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:48.696+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:49.047+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:49.047+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:49.047+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:49.047+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:49.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:49.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:49.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:49.169+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:50.027 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:49.691+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:50.027 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:49.691+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:50.027 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:49.691+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:50.027 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:49.692+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:50.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:50.025+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:50.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:50.025+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:50.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:50.025+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:50.026+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:50.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:50.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:50.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:50.205+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:50.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:50.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:50.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:50.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:50.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:50.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:50.649+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:50.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:50.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:50.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:50.991+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:51.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:51.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:51.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:51.214+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:52.021 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:58:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:58:51.856+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 8998 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:58:52.022 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:51.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:52.022 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:51.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:52.022 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:51.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:52.022 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:51.671+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:52.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:52.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:52.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:52.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:52.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:52.021+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:52.209+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:52.209+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:52.209+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:52.209+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:53.006 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:52.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:53.006 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:52.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:53.006 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:52.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:53.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:52.627+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:53.282 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:53.166+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:53.282 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:53.166+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:53.282 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:53.166+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:53.282 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:53.167+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:53.283 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:53.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:53.283 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:53.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:53.283 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:53.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:53.283 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:53.005+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:53.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:53.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:53.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:53.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:53.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:53.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:53.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:53.632+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:54.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:54.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:54.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:54.038+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:54.187+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:54.187+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:54.187+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:54.188+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:55.013 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:54.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:55.013 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:54.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:55.013 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:54.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:55.013 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:54.648+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:55.012+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:55.012+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:55.012+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:55.013+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:55.221+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:55.221+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:55.221+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:55.221+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:56.051 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:55.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:56.051 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:55.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:56.052 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:55.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:56.052 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:55.617+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:56.050+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:56.050+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:56.050+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:56.051+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:56.185+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:56.185+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:56.185+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:56.186+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:57.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:58:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:58:56.857+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 9060 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:58:57.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:56.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:57.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:56.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:57.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:56.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:57.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:56.621+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:57.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:57.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:57.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:57.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:57.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:57.095+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:57.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:57.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:57.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:57.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:57.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:57.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:57.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:57.204+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:58.065 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:57.618+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:58.065 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:57.618+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:58.065 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:57.618+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:58.065 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:57.619+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:58.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:58.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:58.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:58.243+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:58.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:58.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:58.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:58.064+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:59.068 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:58.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:59.069 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:58.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:59.069 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:58.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:59.069 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:58.591+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:58:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:59.067+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:59.067+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:59.067+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:58:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:58:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:58:59.068+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:59.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:59.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:59.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:58:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:58:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:58:59.284+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:58:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:59.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:59.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:59.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:58:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:58:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:58:59.625+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:00.264 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:00.024+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:00.264 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:00.024+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:00.264 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:00.024+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:00.264 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:00.025+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:00.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:00.263+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:00.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:00.263+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:00.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:00.263+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:00.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:00.263+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:01.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:00.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:01.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:00.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:01.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:00.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:01.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:00.586+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:01.269 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:01.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:01.270 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:01.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:01.270 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:01.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:01.270 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:01.006+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:01.545 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:01.268+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:01.545 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:01.268+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:01.545 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:01.268+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:01.546 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:01.269+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:01.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:01.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:01.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:01.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:01.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:01.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:01.545+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:02.262 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:02.014+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:02.262 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:02.014+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:02.262 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:02.014+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:02.262 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:02.014+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:02.263 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:59:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:59:01.858+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 9122 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:59:02.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:02.260+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:02.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:02.261+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:02.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:02.261+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:02.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:02.261+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:03.021 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:02.579+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:03.021 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:02.579+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:03.021 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:02.579+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:03.021 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:02.580+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:03.221+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:03.221+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:03.221+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:03.222+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:03.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:03.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:03.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:03.020+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:03.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:03.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:03.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:03.532+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:04.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:04.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:04.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:04.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:04.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:04.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:04.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:04.005+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:04.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:04.271+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:04.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:04.271+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:04.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:04.271+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:04.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:04.272+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:04.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:04.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:04.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:04.567+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:05.262 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:04.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:05.262 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:04.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:05.262 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:04.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:05.262 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:04.961+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:05.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:05.261+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:05.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:05.261+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:05.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:05.261+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:05.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:05.261+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:05.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:05.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:05.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:05.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:05.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:05.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:05.569+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:06.243 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:05.932+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:06.243 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:05.932+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:06.243 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:05.932+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:06.243 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:05.933+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:06.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:06.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:06.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:06.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:06.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:06.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:06.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:06.242+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:06.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:06.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:06.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:06.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:06.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:06.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:06.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:06.554+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:07.259 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:59:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:59:06.859+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 9184 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:59:07.259 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:06.959+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:07.259 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:06.959+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:07.259 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:06.959+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:07.259 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:06.960+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:07.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:07.257+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:07.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:07.257+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:07.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:07.257+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:07.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:07.258+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:07.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:07.567+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:07.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:07.567+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:07.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:07.567+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:07.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:07.567+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:08.284 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:08.009+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:08.285 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:08.009+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:08.285 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:08.009+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:08.285 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:08.010+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:08.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:08.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:08.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:08.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:08.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:08.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:08.284+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:08.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:08.524+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:08.524+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:08.524+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:09.245+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:09.245+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:09.245+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:09.245+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:09.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:09.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:09.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:09.003+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:09.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:09.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:09.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:09.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:09.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:09.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:09.533+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:10.269+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:10.269+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:10.269+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:10.270+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:10.048+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:10.048+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:10.048+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:10.049+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:10.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:10.559+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:10.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:10.559+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:10.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:10.559+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:10.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:10.559+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:11.258+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:11.258+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:11.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:11.258+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:11.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:11.258+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:11.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:11.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:11.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:11.017+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:11.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:11.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:11.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:11.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:11.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:11.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:11.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:11.554+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:12.292 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:59:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:59:11.860+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 9246 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:59:12.292 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:11.976+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:12.293 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:11.976+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:12.293 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:11.976+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:12.293 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:11.977+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:12.545 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:12.291+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:12.546 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:12.291+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:12.546 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:12.291+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:12.546 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:12.291+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:12.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:12.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:12.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:12.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:12.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:12.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:12.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:12.545+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:13.291 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:12.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:13.291 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:12.948+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:13.291 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:12.948+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:13.291 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:12.948+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:13.560 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:13.289+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:13.560 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:13.289+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:13.560 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:13.289+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:13.560 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:13.290+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:13.559+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:13.559+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:13.559+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:13.559+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:14.255 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:13.928+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:14.256 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:13.928+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:14.256 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:13.928+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:14.256 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:13.928+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:14.547 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:14.254+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:14.547 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:14.254+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:14.547 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:14.254+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:14.547 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:14.255+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:14.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:14.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:14.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:14.546+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:15.279 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:14.901+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:15.279 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:14.901+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:15.279 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:14.901+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:15.279 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:14.902+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:15.531 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:15.277+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:15.532 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:15.277+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:15.532 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:15.277+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:15.532 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:15.278+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:15.529+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:15.529+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:15.529+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:15.530+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:16.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:15.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:16.274 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:15.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:16.274 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:15.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:16.274 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:15.875+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:16.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:16.271+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:16.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:16.271+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:16.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:16.271+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:16.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:16.271+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:16.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:16.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:16.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:16.567+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:17.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:59:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:59:16.860+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 9308 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:59:17.284 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:16.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:17.284 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:16.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:17.284 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:16.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:17.285 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:16.836+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:17.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:17.282+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:17.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:17.282+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:17.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:17.282+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:17.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:17.282+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:18.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:17.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:18.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:17.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:18.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:17.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:18.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:17.818+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:17.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:17.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:17.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:17.614+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:18.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:18.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:18.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:18.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:18.300+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:18.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:18.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:18.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:18.638+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:19.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:18.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:19.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:18.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:19.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:18.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:19.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:18.836+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:19.263+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:19.263+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:19.263+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:19.263+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:20.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:19.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:20.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:19.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:20.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:19.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:19.641+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:19.848+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:19.848+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:19.848+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:19.848+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:20.284+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:20.284+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:20.284+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:20.285+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:21.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:20.845+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:20.845+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:20.845+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:20.846+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:20.599+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:21.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:20.599+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:21.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:20.599+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:21.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:20.599+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:21.617 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:21.334+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:21.617 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:21.334+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:21.617 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:21.334+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:21.617 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:21.335+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:22.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:59:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:59:21.862+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 9369 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:59:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:21.843+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:21.843+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:21.843+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:21.844+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:21.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:21.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:21.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:21.616+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:22.610 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:22.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:22.610 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:22.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:22.610 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:22.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:22.610 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:22.352+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:22.863 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:22.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:22.863 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:22.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:22.863 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:22.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:22.864 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:22.609+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:23.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:22.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:22.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:22.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:22.863+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:23.618 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:23.338+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:23.618 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:23.338+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:23.618 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:23.338+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:23.618 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:23.339+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:24.000 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:23.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:24.000 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:23.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:24.000 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:23.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:24.000 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:23.617+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:24.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:23.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:24.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:23.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:24.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:23.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:24.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:23.821+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:24.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:24.296+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:24.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:24.296+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:24.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:24.296+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:24.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:24.297+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:25.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:24.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:25.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:24.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:25.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:24.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:25.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:24.655+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:24.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:24.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:24.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:24.804+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:25.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:25.325+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:25.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:25.325+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:25.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:25.325+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:25.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:25.325+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:26.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:25.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:26.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:25.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:25.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:25.648+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:25.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:25.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:25.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:25.817+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:26.645 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:26.348+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:26.645 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:26.348+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:26.645 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:26.348+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:26.645 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:26.348+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:27.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:59:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:59:26.864+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 9431 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:59:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:26.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:26.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:26.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:26.775+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:26.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:26.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:26.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:26.644+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:27.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:27.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:27.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:27.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:27.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:27.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:27.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:27.328+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:28.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:27.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:28.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:27.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:28.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:27.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:28.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:27.671+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:27.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:27.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:27.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:27.797+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:28.663 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:28.362+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:28.664 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:28.362+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:28.664 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:28.362+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:28.664 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:28.362+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:28.662+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:28.662+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:28.662+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:28.663+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:29.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:28.754+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:29.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:28.754+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:29.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:28.754+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:29.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:28.754+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:29.677 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:29.333+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:29.677 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:29.333+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:29.677 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:29.333+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:29.677 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:29.334+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:29.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:29.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:29.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:29.745+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:29.675+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:29.675+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:29.675+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:29.676+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:30.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:30.310+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:30.310+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:30.310+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:30.311+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:30.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:30.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:30.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:30.756+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:30.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:30.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:30.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:30.651+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:31.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:31.316+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:31.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:31.316+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:31.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:31.316+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:31.317+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:32.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:59:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:59:31.865+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 9493 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:59:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:31.802+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:31.802+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:31.802+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:31.802+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:31.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:31.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:31.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:31.670+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:32.687 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:32.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:32.687 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:32.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:32.687 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:32.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:32.687 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:32.354+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:32.814+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:32.814+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:32.814+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:32.815+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:32.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:32.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:32.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:32.686+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:33.579 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:33.317+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:33.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:33.317+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:33.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:33.317+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:33.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:33.318+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:33.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:33.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:33.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:33.698+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:33.826+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:33.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:33.826+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:33.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:33.826+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:33.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:33.827+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:34.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:34.342+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:34.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:34.342+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:34.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:34.342+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:34.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:34.343+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:35.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:34.828+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:34.828+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:34.828+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:34.829+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:34.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:34.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:34.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:34.678+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:35.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:35.375+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:35.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:35.375+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:35.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:35.375+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:35.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:35.376+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:36.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:35.867+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:36.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:35.867+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:35.867+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:35.868+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:35.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:35.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:35.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:35.686+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:36.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:36.395+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:36.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:36.395+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:36.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:36.395+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:36.395+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:36.843 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:36.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:36.843 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:36.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:36.843 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:36.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:36.843 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:36.652+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:36.870 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:36.841+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:36.870 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:36.841+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:36.870 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:36.841+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:36.870 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:36.842+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:37.627 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:59:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:59:36.866+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 9548 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:59:37.627 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:37.356+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:37.627 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:37.356+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:37.627 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:37.356+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:37.627 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:37.356+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:37.811+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:37.811+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:37.811+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:37.812+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:37.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:37.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:37.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:37.626+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:38.807 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:38.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:38.808 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:38.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:38.808 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:38.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:38.808 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:38.582+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:38.808 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:38.395+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:38.808 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:38.395+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:38.808 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:38.395+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:38.808 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:38.396+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:39.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:38.806+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:39.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:38.806+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:39.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:38.806+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:39.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:38.807+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:39.797 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:39.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:39.797 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:39.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:39.797 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:39.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:39.797 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:39.570+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:39.797 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:39.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:39.797 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:39.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:39.797 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:39.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:39.797 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:39.352+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:40.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:39.795+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:40.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:39.795+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:39.795+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:39.796+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:40.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:40.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:40.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:40.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:40.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:40.328+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:41.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:40.844+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:40.844+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:40.844+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:40.845+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:40.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:40.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:40.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:40.601+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:41.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:41.321+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:41.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:41.321+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:41.321+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:41.321+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:41.869 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:41.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:41.869 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:41.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:41.869 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:41.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:41.869 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:41.605+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:41.870 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:41.841+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:41.870 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:41.841+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:41.870 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:41.841+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:41.870 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:41.842+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:42.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:59:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:59:41.868+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 9604 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:59:42.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:42.329+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:42.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:42.329+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:42.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:42.329+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:42.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:42.329+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:42.890 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:42.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:42.890 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:42.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:42.890 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:42.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:42.890 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:42.626+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:43.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:42.889+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:43.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:42.889+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:43.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:42.889+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:42.889+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:43.629 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:43.348+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:43.630 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:43.348+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:43.630 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:43.348+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:43.630 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:43.349+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:43.918 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:43.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:43.918 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:43.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:43.918 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:43.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:43.918 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:43.628+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:44.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:43.916+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:44.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:43.916+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:43.916+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:43.917+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:44.597 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:44.343+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:44.597 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:44.343+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:44.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:44.343+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:44.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:44.344+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:44.905 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:44.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:44.906 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:44.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:44.906 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:44.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:44.906 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:44.614+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:44.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:44.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:44.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:44.905+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:45.638 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:45.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:45.638 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:45.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:45.638 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:45.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:45.638 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:45.390+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:45.952 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:45.636+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:45.952 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:45.636+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:45.952 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:45.636+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:45.952 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:45.637+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:46.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:45.950+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:45.950+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:45.950+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:45.951+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:46.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:46.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:46.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:46.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:46.608+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:46.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:46.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:46.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:46.403+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:47.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:59:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:59:46.869+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 9660 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:59:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:46.945+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:46.945+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:46.945+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:46.945+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:47.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:47.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:47.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:47.589+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:47.398+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:47.398+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:47.398+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:47.399+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:48.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:47.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:48.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:47.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:47.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:47.996+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:48.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:48.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:48.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:48.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:48.625+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:48.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:48.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:48.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:48.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:48.416+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:48.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:48.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:48.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:48.947+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:49.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:49.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:49.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:49.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:49.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:49.616+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:49.369+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:49.369+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:49.369+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:49.370+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:49.901+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:49.901+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:49.901+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:49.902+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:50.651 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:50.329+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:50.651 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:50.329+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:50.651 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:50.329+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:50.651 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:50.330+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:50.932 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:50.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:50.932 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:50.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:50.932 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:50.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:50.932 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:50.650+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:50.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:50.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:50.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:50.931+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:51.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:51.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:51.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:51.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:51.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:51.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:51.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:51.375+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:51.932 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:59:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:59:51.870+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 9716 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:59:51.933 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:51.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:51.933 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:51.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:51.933 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:51.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:51.933 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:51.660+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:52.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:51.931+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:52.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:51.931+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:52.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:51.931+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:51.932+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:52.623 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:52.329+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:52.623 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:52.329+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:52.623 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:52.329+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:52.623 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:52.330+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:52.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:52.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:52.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:52.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:52.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:52.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:52.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:52.621+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:53.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:52.952+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:52.952+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:52.952+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:52.953+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:53.653 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:53.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:53.653 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:53.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:53.653 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:53.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:53.653 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:53.379+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:53.982 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:53.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:53.982 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:53.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:53.982 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:53.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:53.982 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:53.651+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:53.980+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:53.980+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:53.980+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:53.981+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:54.680 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:54.415+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:54.680 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:54.415+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:54.680 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:54.415+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:54.680 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:54.416+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:54.969 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:54.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:54.969 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:54.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:54.969 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:54.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:54.969 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:54.678+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:54.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:54.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:54.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:54.967+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:55.689 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:55.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:55.689 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:55.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:55.689 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:55.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:55.689 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:55.393+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:56.008 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:55.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:56.008 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:55.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:56.008 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:55.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:56.008 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:55.687+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:56.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:56.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:56.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:56.006+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:56.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:56.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:56.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:56.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:56.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:56.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:56.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:56.420+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:57.048 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:56.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:57.049 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:56.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:57.049 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:56.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:57.049 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:56.684+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:57.049 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 03:59:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T03:59:56.870+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 9771 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T03:59:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:57.047+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:57.047+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:57.047+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:57.047+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:57.793 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:57.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:57.793 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:57.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:57.793 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:57.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:57.793 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:57.411+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:57.793 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:57.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:57.794 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:57.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:57.794 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:57.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:57.794 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:57.658+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:58.422 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:58.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:58.422 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:58.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:58.422 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:58.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:58.422 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:58.080+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:58.698 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:58.419+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:58.698 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:58.419+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:58.698 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:58.419+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:58.698 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:58.420+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:59.063 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:58.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:59.064 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:58.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:59.064 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:58.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T03:59:59.064 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:58.697+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T03:59:59.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:59.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:59.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:59.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:59.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:59.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T03:59:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 03:59:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T03:59:59.063+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T03:59:59.713 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:59.434+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:59.714 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:59.434+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:59.714 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:59.434+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T03:59:59.714 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 03:59:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T03:59:59.435+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:00.035 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:59.712+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:00.035 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:59.712+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:00.035 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:59.712+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:00.036 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 03:59:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T03:59:59.713+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:00.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:00.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:00.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:00.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:00.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:00.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:00.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:00.035+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:00.738 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:00.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:00.738 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:00.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:00.738 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:00.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:00.739 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:00.418+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:01.009 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:00.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:01.009 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:00.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:01.009 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:00.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:01.009 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:00.738+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:01.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:01.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:01.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:01.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:01.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:01.008+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:01.765 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:01.390+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:01.766 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:01.390+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:01.766 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:01.390+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:01.766 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:01.391+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:02.023 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:01.763+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:02.024 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:01.763+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:02.024 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:01.763+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:02.024 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:01.764+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:02.024 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:00:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:00:01.872+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 9827 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:00:02.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:02.022+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:02.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:02.022+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:02.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:02.022+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:02.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:02.023+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:02.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:02.357+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:02.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:02.357+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:02.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:02.357+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:02.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:02.358+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:02.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:02.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:02.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:02.988+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:02.810+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:02.810+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:02.810+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:02.811+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:03.767 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:03.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:03.767 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:03.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:03.768 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:03.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:03.768 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:03.374+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:04.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:03.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:04.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:03.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:04.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:03.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:04.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:03.976+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:03.766+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:03.766+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:03.766+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:03.767+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:04.719 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:04.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:04.720 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:04.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:04.720 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:04.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:04.720 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:04.381+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:05.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:04.718+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:05.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:04.718+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:05.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:04.718+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:05.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:04.719+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:05.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:05.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:05.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:05.016+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:05.679 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:05.422+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:05.679 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:05.422+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:05.679 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:05.422+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:05.679 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:05.423+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:06.019 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:05.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:06.019 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:05.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:06.019 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:05.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:06.019 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:05.679+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:06.057+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:06.057+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:06.057+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:06.058+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:06.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:06.471+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:06.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:06.471+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:06.471+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:06.472+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:06.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:06.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:06.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:06.697+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:07.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:00:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:00:06.873+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 9883 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:00:07.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:07.009+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:07.009+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:07.009+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:07.009+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:07.713 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:07.475+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:07.713 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:07.475+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:07.713 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:07.475+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:07.713 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:07.476+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:08.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:07.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:08.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:07.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:07.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:07.735+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:07.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:07.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:07.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:07.981+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:08.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:08.472+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:08.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:08.472+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:08.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:08.472+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:08.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:08.472+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:09.000+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:09.000+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:09.000+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:09.001+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:08.777+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:08.777+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:08.777+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:08.778+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:09.816 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:09.439+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:09.816 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:09.439+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:09.816 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:09.439+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:09.816 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:09.440+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:09.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:09.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:09.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:09.953+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:09.815+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:09.815+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:09.815+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:09.815+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:10.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:10.442+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:10.442+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:10.442+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:10.443+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:11.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:10.918+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:10.918+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:10.918+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:10.919+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:10.855+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:10.855+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:10.855+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:10.856+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:11.455+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:11.455+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:11.455+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:11.455+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:12.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:00:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:00:11.875+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 9940 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:00:12.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:11.881+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:12.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:11.881+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:12.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:11.881+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:12.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:11.882+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:11.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:11.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:11.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:11.931+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:12.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:12.445+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:12.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:12.445+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:12.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:12.445+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:12.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:12.446+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:13.153 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:12.932+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:13.153 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:12.932+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:13.153 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:12.932+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:13.153 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:12.933+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:13.154 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:12.872+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:13.154 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:12.872+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:13.154 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:12.872+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:13.154 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:12.873+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:13.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:13.415+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:13.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:13.415+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:13.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:13.415+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:13.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:13.416+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:14.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:13.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:14.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:13.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:14.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:13.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:14.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:13.960+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:14.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:13.905+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:14.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:13.905+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:14.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:13.905+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:14.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:13.906+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:14.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:14.459+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:14.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:14.459+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:14.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:14.459+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:14.459+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:15.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:14.889+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:15.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:14.889+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:14.889+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:14.889+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:14.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:14.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:14.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:14.916+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:15.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:15.496+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:15.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:15.496+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:15.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:15.496+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:15.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:15.496+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:16.189 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:15.895+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:16.189 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:15.895+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:16.190 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:15.895+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:16.190 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:15.896+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:16.190 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:15.852+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:16.190 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:15.852+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:16.190 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:15.852+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:16.190 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:15.852+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:16.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:16.492+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:16.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:16.492+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:16.492+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:16.493+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:17.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:00:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:00:16.876+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 9996 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:00:17.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:16.844+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:17.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:16.844+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:17.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:16.844+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:17.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:16.845+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:16.851+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:16.851+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:16.851+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:16.851+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:17.818 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:17.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:17.819 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:17.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:17.819 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:17.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:17.819 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:17.529+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:18.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:17.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:18.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:17.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:18.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:17.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:18.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:17.818+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:17.888+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:17.888+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:17.888+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:17.889+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:18.812+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:18.812+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:18.812+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:18.812+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:18.567+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:18.567+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:18.567+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:18.567+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:19.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:18.932+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:19.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:18.932+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:19.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:18.932+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:19.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:18.933+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:19.787 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:19.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:19.788 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:19.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:19.788 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:19.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:19.788 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:19.529+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:19.786+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:19.786+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:19.786+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:19.787+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:19.967+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:19.967+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:19.967+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:19.968+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:20.826 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:20.518+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:20.826 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:20.518+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:20.826 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:20.518+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:20.826 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:20.518+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:21.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:20.962+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:20.962+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:20.962+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:20.963+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:21.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:20.824+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:21.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:20.824+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:21.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:20.824+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:21.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:20.825+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:21.801 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:21.546+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:21.801 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:21.546+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:21.801 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:21.546+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:21.801 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:21.547+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:22.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:21.800+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:22.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:21.800+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:22.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:21.800+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:21.800+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:21.943+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:21.943+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:21.943+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:21.943+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:22.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:00:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:00:21.877+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 10053 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:00:22.786 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:22.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:22.786 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:22.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:22.786 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:22.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:22.786 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:22.517+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:22.785+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:22.785+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:22.785+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:22.786+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:22.961+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:22.961+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:22.961+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:22.962+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:23.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:23.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:23.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:23.742+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:23.559+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:23.559+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:23.559+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:23.560+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:24.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:23.984+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:24.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:23.984+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:24.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:23.984+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:23.984+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:24.992 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:24.789+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:24.993 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:24.789+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:24.993 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:24.789+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:24.993 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:24.790+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:24.993 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:24.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:24.993 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:24.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:24.993 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:24.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:24.993 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:24.594+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:25.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:24.991+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:25.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:24.991+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:25.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:24.991+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:25.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:24.992+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:25.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:25.628+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:25.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:25.628+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:25.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:25.628+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:25.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:25.629+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:26.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:25.943+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:26.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:25.943+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:26.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:25.943+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:25.944+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:25.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:25.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:25.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:25.775+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:27.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:00:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:00:26.878+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 10110 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:00:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:26.895+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:26.895+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:26.895+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:26.896+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:26.656+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:26.656+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:26.656+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:26.656+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:27.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:26.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:27.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:26.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:27.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:26.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:27.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:26.769+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:27.947 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:27.787+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:27.947 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:27.787+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:27.947 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:27.787+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:27.947 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:27.788+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:27.947 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:27.617+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:27.947 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:27.617+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:27.947 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:27.617+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:27.948 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:27.617+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:28.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:27.945+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:28.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:27.945+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:28.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:27.945+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:28.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:27.946+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:28.749+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:28.749+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:28.749+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:28.750+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:28.665+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:28.665+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:28.665+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:28.666+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:29.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:28.992+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:29.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:28.992+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:29.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:28.992+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:29.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:28.993+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:30.038 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:29.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:30.039 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:29.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:30.039 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:29.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:30.039 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:29.710+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:30.039 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:29.617+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:30.039 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:29.617+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:30.039 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:29.617+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:30.039 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:29.618+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:30.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:30.036+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:30.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:30.036+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:30.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:30.036+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:30.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:30.037+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:31.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:30.672+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:31.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:30.672+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:31.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:30.672+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:31.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:30.673+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:31.001 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:30.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:31.001 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:30.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:31.001 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:30.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:31.002 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:30.619+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:31.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:30.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:31.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:30.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:31.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:30.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:31.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:30.999+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:31.720 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:31.612+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:31.721 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:31.612+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:31.721 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:31.612+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:31.721 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:31.613+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:32.030 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:00:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:00:31.878+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 10167 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:00:32.030 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:31.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:32.030 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:31.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:32.030 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:31.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:32.030 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:31.719+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:32.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:32.027+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:32.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:32.027+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:32.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:32.027+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:32.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:32.028+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:32.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:32.671+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:32.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:32.671+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:32.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:32.671+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:32.672+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:32.567+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:32.567+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:32.567+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:32.568+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:33.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:33.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:33.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:33.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:33.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:33.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:33.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:33.075+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:33.695+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:33.695+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:33.695+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:33.695+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:33.537+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:33.537+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:33.537+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:33.538+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:34.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:34.032+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:34.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:34.032+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:34.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:34.032+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:34.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:34.032+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:35.007 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:34.663+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:35.007 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:34.663+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:35.007 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:34.663+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:35.007 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:34.664+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:35.007 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:34.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:35.008 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:34.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:35.008 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:34.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:35.008 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:34.579+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:35.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:35.004+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:35.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:35.004+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:35.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:35.004+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:35.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:35.005+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:36.021 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:35.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:36.021 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:35.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:36.021 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:35.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:36.021 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:35.711+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:36.021 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:35.582+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:36.021 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:35.582+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:36.022 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:35.582+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:36.022 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:35.582+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:36.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:36.019+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:36.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:36.019+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:36.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:36.019+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:36.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:36.020+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:36.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:36.706+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:36.706+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:36.706+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:36.706+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:36.615+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:36.615+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:36.615+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:36.616+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:37.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:00:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:00:36.881+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 10223 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:00:37.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:37.051+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:37.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:37.051+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:37.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:37.051+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:37.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:37.052+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:37.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:37.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:37.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:37.714+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:37.648+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:37.648+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:37.648+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:37.649+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:38.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:38.095+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:38.095+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:38.095+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:38.096+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:39.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:38.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:39.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:38.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:39.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:38.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:38.665+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:38.674+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:38.674+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:38.674+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:38.674+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:39.457 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:39.145+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:39.457 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:39.145+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:39.457 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:39.145+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:39.457 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:39.145+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:39.731 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:39.668+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:39.731 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:39.668+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:39.731 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:39.668+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:39.731 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:39.669+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:39.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:39.681+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:39.732 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:39.681+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:39.732 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:39.681+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:39.732 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:39.682+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:40.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:40.100+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:40.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:40.100+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:40.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:40.100+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:40.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:40.101+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:41.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:40.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:40.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:40.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:40.714+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:41.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:40.692+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:41.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:40.692+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:41.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:40.692+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:41.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:40.692+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:41.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:41.143+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:41.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:41.143+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:41.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:41.143+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:41.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:41.144+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:41.883 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:41.748+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:41.883 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:41.748+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:41.884 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:41.748+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:41.884 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:41.749+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:41.884 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:41.711+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:41.884 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:41.711+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:41.884 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:41.711+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:41.884 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:41.712+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:42.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:00:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:00:41.882+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 10280 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:00:42.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:42.094+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:42.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:42.094+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:42.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:42.094+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:42.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:42.095+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:43.056 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:42.784+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:43.056 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:42.784+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:43.056 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:42.784+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:43.057 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:42.784+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:43.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:42.689+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:43.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:42.689+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:43.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:42.689+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:43.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:42.690+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:43.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:43.055+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:43.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:43.055+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:43.055+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:43.056+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:43.872 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:43.653+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:43.872 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:43.653+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:43.873 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:43.653+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:43.873 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:43.654+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:43.873 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:43.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:43.873 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:43.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:43.873 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:43.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:43.873 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:43.747+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:44.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:44.093+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:44.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:44.093+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:44.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:44.093+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:44.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:44.093+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:45.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:44.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:45.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:44.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:44.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:44.758+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:45.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:44.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:45.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:44.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:45.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:44.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:45.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:44.639+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:45.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:45.094+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:45.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:45.094+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:45.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:45.094+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:45.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:45.095+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:46.069 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:45.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:46.069 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:45.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:46.069 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:45.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:46.070 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:45.725+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:46.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:45.686+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:46.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:45.686+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:46.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:45.686+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:46.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:45.686+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:46.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:46.068+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:46.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:46.068+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:46.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:46.068+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:46.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:46.068+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:47.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:00:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:00:46.883+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 10337 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:00:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:46.762+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:46.762+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:46.762+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:46.763+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:46.709+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:46.709+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:46.709+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:46.709+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:47.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:47.084+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:47.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:47.084+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:47.084+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:47.085+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:48.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:47.764+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:48.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:47.764+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:47.764+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:47.765+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:47.746+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:47.746+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:47.746+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:47.747+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:48.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:48.095+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:48.095+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:48.095+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:48.096+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:49.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:48.780+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:49.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:48.780+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:48.780+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:48.781+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:48.737+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:48.737+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:48.737+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:48.738+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:49.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:49.128+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:49.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:49.128+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:49.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:49.128+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:49.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:49.129+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:50.017 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:49.741+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:50.017 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:49.741+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:50.017 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:49.741+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:50.017 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:49.741+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:50.018 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:49.780+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:50.018 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:49.780+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:50.018 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:49.780+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:50.018 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:49.781+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:50.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:50.093+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:50.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:50.093+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:50.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:50.093+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:50.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:50.094+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:51.051 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:50.706+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:51.051 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:50.706+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:51.051 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:50.706+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:51.051 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:50.707+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:51.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:50.784+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:51.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:50.784+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:51.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:50.784+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:51.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:50.785+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:51.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:51.049+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:51.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:51.049+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:51.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:51.049+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:51.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:51.050+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:52.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:00:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:00:51.884+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 10393 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:00:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:51.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:51.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:51.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:51.745+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:51.806+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:52.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:51.806+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:52.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:51.806+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:52.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:51.806+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:52.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:52.091+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:52.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:52.091+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:52.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:52.091+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:52.092+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:53.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:52.760+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:52.760+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:52.760+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:52.760+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:52.829+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:52.829+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:52.829+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:52.829+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:53.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:53.110+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:53.599 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:53.110+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:53.599 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:53.110+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:53.600 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:53.110+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:54.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:53.805+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:53.805+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:53.805+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:53.805+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:53.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:53.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:53.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:53.717+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:54.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:54.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:54.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:54.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:54.097+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:55.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:54.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:55.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:54.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:55.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:54.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:55.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:54.746+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:55.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:54.831+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:55.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:54.831+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:55.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:54.831+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:55.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:54.832+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:55.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:55.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:55.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:55.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:55.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:55.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:55.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:55.133+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:56.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:55.743+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:56.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:55.743+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:56.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:55.743+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:56.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:55.743+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:55.837+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:55.837+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:55.837+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:55.837+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:56.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:56.167+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:56.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:56.167+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:56.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:56.167+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:56.167+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:57.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:00:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:00:56.885+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 10447 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:00:57.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:56.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:56.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:56.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:56.723+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:56.867+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:56.867+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:56.867+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:56.868+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:57.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:57.217+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:57.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:57.217+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:57.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:57.217+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:57.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:57.218+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:58.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:57.761+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:57.761+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:57.761+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:57.762+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:57.891+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:57.891+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:57.891+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:57.891+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:58.183+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:58.183+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:58.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:58.183+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:58.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:58.184+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:00:59.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:58.903+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:59.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:58.903+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:58.903+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:00:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:58.904+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:58.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:58.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:58.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:00:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:58.737+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:00:59.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:59.217+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:59.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:59.217+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:59.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:59.217+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:00:59.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:00:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:00:59.218+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:00.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:59.782+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:00.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:59.782+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:00.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:59.782+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:00.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:00:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:00:59.783+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:00.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:59.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:00.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:59.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:00.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:59.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:00.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:00:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:00:59.928+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:00.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:00.196+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:00.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:00.196+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:00.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:00.196+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:00.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:00.197+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:01.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:00.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:01.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:00.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:01.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:00.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:00.920+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:00.762+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:00.762+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:00.762+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:00.763+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:01.505 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:01.246+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:01.505 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:01.246+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:01.505 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:01.246+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:01.505 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:01.247+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:01.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:01.723+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:01.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:01.723+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:01.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:01.723+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:01.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:01.724+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:02.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:01.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:02.261 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:01.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:02.261 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:01.956+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:02.261 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:01.956+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:02.261 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:01:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:01:01.886+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 10503 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:01:02.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:02.259+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:02.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:02.259+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:02.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:02.259+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:02.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:02.260+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:03.006 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:02.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:03.007 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:02.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:03.007 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:02.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:03.007 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:02.725+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:03.275 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:03.005+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:03.275 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:03.005+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:03.275 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:03.005+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:03.275 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:03.006+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:03.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:03.273+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:03.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:03.273+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:03.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:03.273+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:03.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:03.274+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:04.022 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:03.680+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:04.022 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:03.680+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:04.022 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:03.680+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:04.022 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:03.681+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:04.324 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:04.021+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:04.325 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:04.021+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:04.325 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:04.021+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:04.325 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:04.021+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:04.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:04.323+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:04.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:04.323+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:04.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:04.323+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:04.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:04.324+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:04.986 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:04.636+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:04.986 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:04.636+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:04.986 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:04.636+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:04.986 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:04.637+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:05.302 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:04.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:05.302 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:04.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:05.302 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:04.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:05.303 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:04.985+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:05.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:05.301+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:05.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:05.301+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:05.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:05.301+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:05.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:05.302+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:05.949 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:05.632+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:05.949 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:05.632+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:05.949 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:05.632+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:05.949 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:05.633+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:06.325 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:05.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:06.325 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:05.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:06.325 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:05.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:06.325 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:05.948+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:06.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:06.323+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:06.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:06.323+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:06.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:06.323+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:06.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:06.324+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:06.977 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:01:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:01:06.888+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 10559 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:01:06.978 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:06.662+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:06.978 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:06.662+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:06.978 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:06.662+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:06.978 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:06.663+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:07.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:06.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:07.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:06.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:07.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:06.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:07.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:06.975+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:07.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:07.327+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:07.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:07.327+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:07.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:07.327+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:07.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:07.328+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:07.998 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:07.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:07.999 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:07.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:07.999 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:07.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:07.999 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:07.626+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:07.996+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:08.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:07.996+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:08.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:07.996+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:08.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:07.997+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:08.620 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:08.346+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:08.620 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:08.346+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:08.620 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:08.346+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:08.620 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:08.346+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:09.003 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:08.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:09.003 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:08.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:09.004 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:08.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:09.004 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:08.618+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:09.321 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:09.001+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:09.321 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:09.001+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:09.321 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:09.001+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:09.321 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:09.002+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:09.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:09.319+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:09.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:09.319+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:09.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:09.319+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:09.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:09.319+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:09.996 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:09.579+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:09.996 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:09.579+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:09.996 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:09.579+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:09.996 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:09.580+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:09.994+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:09.994+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:09.994+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:09.994+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:10.632 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:10.331+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:10.632 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:10.331+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:10.632 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:10.331+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:10.632 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:10.332+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:11.040 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:10.629+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:11.041 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:10.629+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:11.041 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:10.629+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:11.041 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:10.630+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:11.303 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:11.038+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:11.303 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:11.038+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:11.303 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:11.038+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:11.303 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:11.039+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:11.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:11.301+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:11.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:11.301+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:11.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:11.301+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:11.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:11.302+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:11.890 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:11.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:11.890 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:11.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:11.891 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:11.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:11.891 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:11.621+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:12.166 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:12.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:12.166 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:12.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:12.166 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:12.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:12.166 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:12.060+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:12.167 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:01:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:01:11.888+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 10615 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:01:12.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:12.286+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:12.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:12.286+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:12.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:12.286+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:12.287+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:13.074 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:12.632+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:13.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:12.632+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:13.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:12.632+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:13.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:12.633+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:13.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:13.323+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:13.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:13.323+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:13.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:13.323+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:13.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:13.323+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:13.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:13.073+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:13.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:13.073+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:13.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:13.073+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:13.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:13.073+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:14.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:13.653+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:14.090 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:13.653+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:14.090 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:13.653+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:14.090 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:13.654+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:14.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:14.282+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:14.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:14.282+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:14.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:14.282+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:14.283+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:14.114+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:14.114+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:14.114+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:14.115+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:15.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:14.675+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:15.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:14.675+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:15.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:14.675+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:15.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:14.676+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:15.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:15.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:15.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:15.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:15.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:15.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:15.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:15.239+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:15.164+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:15.164+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:15.164+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:15.165+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:16.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:15.640+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:16.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:15.640+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:16.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:15.640+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:15.641+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:16.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:16.157+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:16.157+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:16.157+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:16.158+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:16.207+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:16.207+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:16.207+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:16.208+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:16.891 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:16.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:16.891 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:16.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:16.892 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:16.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:16.892 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:16.627+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:17.191 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:01:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:01:16.890+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 10670 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:01:17.192 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:17.109+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:17.192 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:17.109+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:17.192 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:17.109+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:17.192 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:17.110+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:17.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:17.190+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:17.190+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:17.190+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:17.191+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:18.071 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:17.645+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:18.071 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:17.645+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:18.071 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:17.645+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:18.071 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:17.645+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:18.069+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:18.069+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:18.069+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:18.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:18.070+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:18.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:18.203+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:18.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:18.203+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:18.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:18.203+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:18.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:18.204+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:18.695+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:18.695+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:18.695+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:19.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:18.696+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:19.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:19.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:19.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:19.081+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:19.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:19.225+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:19.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:19.225+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:19.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:19.225+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:19.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:19.225+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:19.707+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:19.707+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:19.707+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:19.707+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:20.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:20.102+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:20.102+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:20.102+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:20.103+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:20.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:20.178+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:20.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:20.178+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:20.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:20.178+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:20.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:20.179+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:21.079 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:20.685+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:21.079 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:20.685+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:21.079 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:20.685+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:21.079 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:20.685+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:21.078+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:21.078+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:21.078+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:21.079+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:21.161+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:21.161+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:21.161+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:21.162+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:22.038 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:01:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:01:21.891+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 10726 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:01:22.038 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:21.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:22.038 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:21.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:22.038 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:21.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:22.038 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:21.728+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:22.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:22.211+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:22.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:22.211+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:22.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:22.211+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:22.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:22.212+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:22.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:22.036+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:22.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:22.036+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:22.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:22.036+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:22.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:22.037+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:23.014 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:22.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:23.014 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:22.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:23.015 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:22.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:23.015 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:22.756+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:23.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:23.246+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:23.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:23.246+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:23.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:23.246+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:23.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:23.247+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:23.013+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:23.013+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:23.013+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:23.014+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:24.013 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:23.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:24.013 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:23.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:24.013 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:23.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:24.013 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:23.758+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:24.012+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:24.012+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:24.012+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:24.012+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:24.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:24.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:24.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:24.240+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:25.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:24.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:25.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:24.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:25.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:24.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:25.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:24.800+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:25.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:24.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:25.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:24.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:25.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:24.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:25.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:25.000+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:25.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:25.253+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:25.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:25.253+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:25.253+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:25.254+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:26.296 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:25.834+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:26.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:25.834+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:26.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:25.834+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:26.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:25.835+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:26.297 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:25.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:26.297 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:25.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:26.297 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:25.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:26.297 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:25.972+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:26.562 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:26.295+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:26.562 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:26.295+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:26.562 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:26.295+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:26.562 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:26.296+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:27.329 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:26.887+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:27.329 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:26.887+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:27.329 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:26.887+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:27.329 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:26.887+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:27.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:01:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:01:26.892+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 10782 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:01:27.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:27.021+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:27.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:27.021+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:27.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:27.021+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:27.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:27.021+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:27.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:27.327+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:27.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:27.327+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:27.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:27.327+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:27.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:27.328+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:28.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:27.841+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:28.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:27.841+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:28.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:27.841+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:28.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:27.842+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:28.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:28.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:28.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:28.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:28.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:28.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:28.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:28.056+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:28.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:28.319+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:28.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:28.319+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:28.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:28.319+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:28.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:28.320+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:29.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:28.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:29.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:28.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:28.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:28.818+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:29.371 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:29.095+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:29.371 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:29.095+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:29.371 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:29.095+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:29.371 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:29.096+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:29.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:29.369+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:29.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:29.369+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:29.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:29.369+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:29.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:29.370+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:30.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:29.838+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:30.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:29.838+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:30.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:29.838+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:29.839+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:30.061+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:30.061+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:30.061+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:30.062+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:30.797 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:30.392+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:30.797 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:30.392+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:30.797 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:30.392+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:30.797 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:30.392+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:30.795+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:30.795+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:30.795+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:30.796+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:31.039+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:31.039+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:31.039+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:31.040+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:31.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:31.440+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:31.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:31.440+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:31.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:31.440+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:31.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:31.441+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:32.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:01:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:01:31.893+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 10838 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:01:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:31.767+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:31.767+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:31.767+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:31.768+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:32.443 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:32.090+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:32.443 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:32.090+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:32.443 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:32.090+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:32.443 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:32.091+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:32.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:32.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:32.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:32.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:32.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:32.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:32.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:32.442+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:32.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:32.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:32.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:32.772+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:33.481 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:33.084+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:33.482 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:33.084+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:33.482 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:33.084+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:33.482 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:33.085+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:33.750 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:33.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:33.750 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:33.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:33.750 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:33.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:33.750 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:33.481+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:34.061 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:33.748+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:34.061 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:33.748+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:34.061 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:33.748+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:34.062 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:33.749+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:34.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:34.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:34.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:34.061+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:34.796 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:34.490+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:34.796 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:34.490+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:34.796 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:34.490+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:34.796 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:34.491+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:35.078 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:34.794+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:35.078 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:34.794+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:35.078 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:34.794+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:35.078 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:34.795+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:35.076+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:35.076+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:35.076+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:35.077+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:35.814 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:35.538+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:35.814 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:35.538+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:35.814 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:35.538+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:35.814 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:35.539+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:36.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:35.813+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:36.076 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:35.813+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:36.076 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:35.813+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:36.076 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:35.814+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:36.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:36.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:36.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:36.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:36.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:36.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:36.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:36.075+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:36.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:36.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:36.803 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:36.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:36.803 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:36.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:36.803 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:36.497+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:37.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:01:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:01:36.895+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 10893 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:01:37.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:36.801+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:36.801+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:36.801+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:36.802+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:37.526 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:37.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:37.527 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:37.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:37.527 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:37.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:37.527 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:37.109+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:37.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:37.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:37.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:37.525+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:37.761+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:37.762+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:37.762+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:37.762+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:38.481 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:38.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:38.482 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:38.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:38.482 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:38.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:38.482 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:38.103+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:38.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:38.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:38.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:38.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:38.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:38.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:38.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:38.481+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:38.721+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:38.721+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:38.721+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:38.721+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:39.059+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:39.059+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:39.059+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:39.059+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:39.685 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:39.432+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:39.685 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:39.432+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:39.685 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:39.432+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:39.685 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:39.433+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:40.030 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:39.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:40.030 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:39.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:40.030 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:39.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:40.030 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:39.684+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:40.028+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:40.028+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:40.028+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:40.029+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:40.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:40.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:40.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:40.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:40.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:40.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:40.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:40.460+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:41.003 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:40.721+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:41.003 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:40.721+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:41.003 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:40.721+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:41.003 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:40.722+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:41.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:41.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:41.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:41.002+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:41.714 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:41.505+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:41.714 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:41.505+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:41.714 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:41.505+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:41.714 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:41.506+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:42.011 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:01:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:01:41.896+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 10949 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:01:42.011 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:41.712+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:42.011 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:41.712+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:42.012 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:41.712+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:42.012 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:41.713+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:42.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:42.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:42.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:42.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:42.010+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:42.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:42.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:42.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:42.666+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:42.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:42.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:42.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:42.510+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:43.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:42.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:42.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:42.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:42.976+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:43.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:43.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:43.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:43.618+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:43.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:43.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:43.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:43.556+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:43.982+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:43.982+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:43.982+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:43.982+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:44.946 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:44.601+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:44.946 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:44.601+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:44.947 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:44.601+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:44.947 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:44.602+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:44.947 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:44.590+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:44.947 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:44.590+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:44.947 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:44.590+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:44.947 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:44.591+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:45.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:44.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:45.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:44.944+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:45.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:44.944+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:45.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:44.945+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:45.924 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:45.606+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:45.924 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:45.606+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:45.925 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:45.606+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:45.925 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:45.607+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:45.925 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:45.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:45.925 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:45.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:45.925 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:45.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:45.926 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:45.587+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:46.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:45.921+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:46.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:45.921+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:46.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:45.921+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:46.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:45.922+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:46.897 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:46.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:46.897 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:46.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:46.897 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:46.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:46.897 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:46.580+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:46.898 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:46.642+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:46.898 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:46.642+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:46.898 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:46.642+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:46.898 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:46.642+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:47.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:01:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:01:46.897+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11004 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:01:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:46.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:46.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:46.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:46.895+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:47.676+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:47.676+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:47.676+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:47.677+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:47.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:47.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:47.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:47.536+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:48.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:47.887+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:47.887+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:47.887+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:47.888+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:48.634+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:48.634+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:48.634+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:48.635+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:48.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:48.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:48.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:48.500+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:48.933+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:48.933+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:48.933+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:48.934+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:49.624+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:49.624+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:49.624+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:49.624+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:49.453+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:49.453+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:49.453+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:49.453+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:49.891+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:49.891+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:49.891+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:49.891+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:50.674+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:50.674+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:50.674+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:50.675+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:50.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:50.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:50.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:50.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:50.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:50.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:50.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:50.497+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:50.907+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:50.907+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:50.907+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:50.908+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:51.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:51.689+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:51.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:51.689+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:51.689+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:51.690+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:51.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:51.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:51.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:51.516+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:52.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:01:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:01:51.899+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11060 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:01:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:51.953+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:51.953+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:51.953+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:51.954+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:52.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:52.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:52.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:52.718+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:52.536+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:52.536+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:52.536+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:52.537+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:52.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:52.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:52.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:52.930+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:53.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:53.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:53.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:53.670+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:53.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:53.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:53.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:53.497+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:53.921+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:53.921+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:53.921+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:53.922+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:54.671+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:54.671+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:54.671+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:54.672+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:54.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:54.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:54.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:54.522+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:54.907+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:54.907+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:54.907+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:54.908+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:55.635+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:55.635+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:55.635+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:55.636+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:55.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:55.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:55.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:55.499+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:55.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:55.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:56.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:55.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:55.955+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:56.661+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:56.661+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:56.661+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:56.661+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:56.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:56.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:56.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:56.547+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:57.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:01:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:01:56.900+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11115 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:01:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:56.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:56.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:56.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:56.917+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:57.895 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:57.628+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:57.895 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:57.628+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:57.895 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:57.628+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:57.895 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:57.629+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:57.895 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:57.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:57.895 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:57.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:57.895 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:57.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:57.896 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:57.589+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:57.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:57.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:57.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:57.894+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:58.846 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:58.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:58.846 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:58.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:58.847 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:58.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:58.847 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:58.666+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:58.847 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:58.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:58.847 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:58.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:58.847 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:58.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:58.847 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:58.581+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:58.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:58.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:58.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:01:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:58.846+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:01:59.872 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:59.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:59.872 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:59.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:59.872 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:59.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:01:59.872 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:01:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:01:59.582+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:01:59.872 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:59.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:59.872 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:59.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:59.872 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:59.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:01:59.873 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:01:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:01:59.660+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:59.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:59.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:59.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:01:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:01:59.871+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:00.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:00.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:00.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:00.622+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:00.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:00.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:00.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:00.561+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:00.831+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:00.831+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:00.831+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:00.832+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:01.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:01.563+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:01.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:01.563+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:01.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:01.563+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:01.563+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:01.671+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:01.671+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:01.671+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:01.672+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:02.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:02:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:02:01.901+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11171 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:02:02.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:01.838+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:01.838+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:01.838+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:01.839+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:02.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:02.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:02.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:02.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:02.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:02.548+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:02.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:02.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:02.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:02.719+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:02.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:02.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:02.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:02.879+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:03.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:03.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:03.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:03.510+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:03.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:03.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:03.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:03.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:03.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:03.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:03.735+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:03.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:03.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:03.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:03.889+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:04.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:04.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:04.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:04.688+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:04.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:04.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:04.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:04.486+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:04.872+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:04.872+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:04.872+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:04.873+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:05.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:05.469+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:05.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:05.469+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:05.735 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:05.469+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:05.735 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:05.470+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:06.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:05.733+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:06.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:05.733+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:06.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:05.733+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:05.734+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:05.851+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:05.851+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:05.851+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:05.852+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:06.818 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:06.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:06.818 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:06.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:06.818 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:06.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:06.818 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:06.455+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:06.819 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:06.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:06.819 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:06.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:06.819 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:06.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:06.819 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:06.688+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:07.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:02:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:02:06.902+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11227 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:02:07.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:06.859+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:07.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:06.859+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:07.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:06.859+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:07.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:06.860+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:07.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:07.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:07.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:07.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:07.497+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:07.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:07.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:07.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:07.711+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:08.179 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:07.907+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:08.179 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:07.907+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:08.179 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:07.907+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:08.179 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:07.908+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:08.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:08.529+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:08.529+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:08.529+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:08.529+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:08.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:08.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:08.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:08.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:08.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:08.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:08.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:08.726+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:08.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:08.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:09.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:08.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:09.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:08.898+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:09.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:09.739+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:09.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:09.739+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:09.739+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:09.740+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:09.543+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:09.543+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:09.543+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:09.544+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:09.883+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:09.883+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:09.883+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:09.883+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:10.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:10.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:10.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:10.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:10.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:10.723+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:10.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:10.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:10.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:10.534+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:10.837+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:10.837+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:10.837+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:11.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:10.837+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:11.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:11.700+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:11.700+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:11.700+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:11.701+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:11.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:11.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:11.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:11.570+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:11.802+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:11.802+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:11.802+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:11.802+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:12.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:02:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:02:11.904+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11283 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:02:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:12.737+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:12.737+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:12.737+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:12.738+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:12.528+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:12.528+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:12.528+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:12.529+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:13.262 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:12.839+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:13.262 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:12.839+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:13.262 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:12.839+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:13.262 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:12.840+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:13.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:13.508+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:13.508+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:13.508+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:13.509+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:13.847 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:13.737+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:13.847 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:13.737+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:13.847 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:13.737+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:13.847 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:13.738+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:13.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:13.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:13.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:13.846+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:14.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:14.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:14.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:14.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:14.723+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:14.541+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:14.541+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:14.541+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:14.542+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:14.891+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:14.891+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:14.891+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:14.892+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:15.694 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:15.524+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:15.694 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:15.524+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:15.694 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:15.524+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:15.694 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:15.524+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:16.032 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:15.692+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:16.032 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:15.692+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:16.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:15.692+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:16.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:15.693+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:16.033 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:15.903+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:16.033 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:15.903+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:16.033 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:15.903+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:16.033 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:15.903+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:16.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:16.694+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:16.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:16.694+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:16.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:16.694+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:16.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:16.695+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:16.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:16.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:16.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:16.542+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:17.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:02:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:02:16.905+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11339 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:02:17.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:16.901+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:17.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:16.901+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:17.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:16.901+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:17.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:16.902+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:17.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:17.706+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:17.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:17.706+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:17.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:17.706+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:17.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:17.706+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:17.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:17.560+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:17.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:17.560+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:17.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:17.560+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:17.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:17.561+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:17.860+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:17.860+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:17.860+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:18.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:17.860+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:18.743 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:18.519+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:18.743 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:18.519+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:18.743 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:18.519+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:18.743 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:18.520+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:18.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:18.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:18.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:18.743+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:18.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:18.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:18.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:18.898+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:19.786+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:19.786+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:19.786+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:19.786+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:19.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:19.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:19.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:19.552+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:20.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:19.883+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:20.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:19.883+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:20.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:19.883+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:19.884+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:20.750+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:20.750+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:20.750+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:20.750+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:20.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:20.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:20.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:20.534+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:20.853+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:20.853+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:20.853+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:20.854+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:21.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:21.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:21.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:21.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:21.767+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:21.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:21.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:21.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:21.543+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:22.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:21.842+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:22.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:21.842+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:22.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:21.842+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:22.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:21.843+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:22.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:02:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:02:21.905+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11395 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:02:22.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:22.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:22.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:22.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:22.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:22.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:22.739+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:22.511+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:22.511+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:22.511+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:22.511+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:23.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:22.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:23.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:22.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:22.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:22.864+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:23.712 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:23.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:23.712 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:23.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:23.712 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:23.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:23.712 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:23.556+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:24.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:23.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:24.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:23.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:23.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:23.711+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:24.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:23.846+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:24.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:23.846+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:24.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:23.846+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:24.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:23.847+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:24.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:24.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:24.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:24.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:24.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:24.665+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:24.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:24.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:24.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:24.535+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:25.248 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:24.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:25.248 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:24.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:25.248 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:24.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:25.249 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:24.890+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:25.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:25.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:25.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:25.711+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:25.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:25.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:25.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:25.567+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:26.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:25.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:26.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:25.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:26.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:25.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:26.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:25.917+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:26.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:26.678+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:26.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:26.678+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:26.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:26.678+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:26.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:26.679+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:26.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:26.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:26.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:26.578+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:27.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:02:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:02:26.906+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11452 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:02:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:26.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:26.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:26.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:26.930+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:27.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:27.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:27.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:27.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:27.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:27.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:27.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:27.600+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:27.895 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:27.719+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:27.895 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:27.719+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:27.895 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:27.719+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:27.895 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:27.719+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:28.321 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:27.892+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:28.321 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:27.892+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:28.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:27.892+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:28.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:27.893+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:28.705+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:28.705+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:28.705+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:28.706+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:28.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:28.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:28.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:28.586+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:29.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:28.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:29.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:28.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:29.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:28.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:29.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:28.932+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:29.940 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:29.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:29.940 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:29.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:29.940 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:29.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:29.940 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:29.584+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:29.941 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:29.754+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:29.941 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:29.754+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:29.941 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:29.754+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:29.941 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:29.755+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:30.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:29.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:30.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:29.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:30.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:29.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:29.940+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:30.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:30.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:30.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:30.776+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:30.536+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:30.536+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:30.536+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:30.537+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:30.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:30.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:30.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:30.974+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:31.754 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:31.494+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:31.755 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:31.494+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:31.755 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:31.494+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:31.755 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:31.495+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:32.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:02:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:02:31.909+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11509 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:02:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:31.753+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:31.753+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:31.753+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:31.754+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:31.980+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:31.980+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:31.980+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:31.981+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:32.804 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:32.740+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:32.805 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:32.740+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:32.805 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:32.740+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:32.805 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:32.741+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:32.805 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:32.536+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:32.805 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:32.536+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:32.805 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:32.536+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:32.805 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:32.537+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:33.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:32.959+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:33.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:32.959+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:33.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:32.959+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:33.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:32.960+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:33.758+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:33.758+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:33.758+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:33.759+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:33.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:33.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:33.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:33.572+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:33.951+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:33.951+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:33.951+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:33.951+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:34.809 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:34.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:34.810 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:34.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:34.810 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:34.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:34.810 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:34.535+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:35.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:34.808+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:34.808+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:34.808+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:34.809+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:35.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:34.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:35.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:34.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:35.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:34.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:35.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:34.942+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:35.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:35.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:35.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:35.518+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:36.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:35.855+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:36.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:35.855+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:36.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:35.855+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:36.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:35.855+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:35.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:35.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:35.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:35.931+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:36.672 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T04:02:36.669+0000 7f1a2493f640 0 monclient(hunting): authenticate timed out after 300 2024-09-10T04:02:36.673 INFO:teuthology.orchestra.run.smithi112.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-09-10T04:02:36.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:36.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:36.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:36.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:36.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:36.549+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:37.144 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:02:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:02:36.910+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11565 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:02:37.144 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:36.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:37.144 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:36.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:37.144 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:36.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:37.144 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:36.975+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:37.145 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:36.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:37.145 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:36.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:37.145 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:36.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:37.145 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:36.872+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:37.574 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -e sha1=3a5e3f16720f432aea965c40a8f6eb480095ec6c -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-09-10T04:02:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:37.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:37.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:37.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:37.511+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:37.837 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T04:02:38.174 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:37.911+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:38.175 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:37.911+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:38.175 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:37.911+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:38.175 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:37.911+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:38.175 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:37.936+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:38.175 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:37.936+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:38.175 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:37.936+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:38.175 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:37.937+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:38.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:38.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:38.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:38.504+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:38.868+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:38.868+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:38.868+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:38.869+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:38.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:38.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:38.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:38.944+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:39.472+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:39.472+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:39.473+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:39.473+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:40.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:39.898+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:40.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:39.898+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:39.898+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:39.899+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:39.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:39.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:39.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:39.979+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:40.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:40.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:40.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:40.446+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:40.936+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:40.936+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:40.936+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:40.937+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:40.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:40.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:40.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:40.895+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:41.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:41.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:41.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:41.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:41.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:41.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:41.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:41.398+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:42.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:02:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:02:41.911+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11622 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:02:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:41.919+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:41.919+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:41.919+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:41.920+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:41.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:42.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:41.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:42.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:41.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:42.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:41.917+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:42.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:42.353+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:42.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:42.353+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:42.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:42.353+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:42.354+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:43.100 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:42.876+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:43.100 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:42.876+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:43.100 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:42.876+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:43.100 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:42.877+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:43.100 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:42.917+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:43.100 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:42.917+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:43.100 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:42.917+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:43.101 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:42.918+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:43.361 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:43.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:43.361 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:43.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:43.361 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:43.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:43.361 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:43.313+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:44.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:43.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:44.310 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:43.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:44.310 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:43.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:44.310 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:43.959+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:44.310 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:43.924+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:44.310 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:43.924+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:44.310 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:43.924+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:44.310 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:43.924+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:44.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:44.308+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:44.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:44.308+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:44.308+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:44.309+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:45.328 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:44.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:45.328 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:44.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:45.328 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:44.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:45.328 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:44.982+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:45.329 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:44.933+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:45.329 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:44.933+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:45.329 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:44.933+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:45.329 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:44.933+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:45.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:45.326+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:45.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:45.326+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:45.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:45.326+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:45.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:45.327+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:46.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:45.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:45.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:45.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:45.932+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:46.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:46.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:46.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:46.027+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:46.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:46.373+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:46.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:46.373+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:46.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:46.373+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:46.374+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:47.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:47.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:47.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:47.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:47.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:47.050+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:46.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:46.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:46.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:46.900+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:47.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:02:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:02:46.912+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11679 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:02:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:47.364+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:47.364+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:47.364+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:47.364+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:48.323 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:48.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:48.323 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:48.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:48.323 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:48.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:48.323 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:48.019+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:48.323 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:47.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:48.323 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:47.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:48.324 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:47.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:48.324 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:47.880+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:48.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:48.321+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:48.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:48.321+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:48.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:48.321+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:48.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:48.322+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:49.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:49.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:49.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:49.034+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:48.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:48.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:48.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:48.919+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:49.825 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:49.335+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:49.825 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:49.335+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:49.825 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:49.335+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:49.825 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:49.335+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:50.320 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:50.052+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:50.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:50.052+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:50.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:50.052+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:50.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:50.053+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:50.321 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:49.901+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:50.321 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:49.901+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:50.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:49.901+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:50.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:49.901+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:50.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:50.318+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:50.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:50.318+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:50.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:50.318+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:50.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:50.319+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:51.095+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:51.095+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:51.095+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:51.095+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:50.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:50.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:50.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:50.918+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:51.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:51.343+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:51.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:51.343+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:51.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:51.343+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:51.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:51.344+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:52.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:02:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:02:51.914+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11736 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:02:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:52.065+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:52.065+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:52.065+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:52.066+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:51.874+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:51.874+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:51.874+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:51.875+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:52.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:52.345+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:52.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:52.345+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:52.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:52.345+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:52.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:52.345+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:53.117 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:52.840+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:53.117 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:52.840+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:53.117 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:52.840+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:53.118 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:52.840+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:53.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:53.115+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:53.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:53.115+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:53.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:53.115+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:53.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:53.116+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:53.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:53.315+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:53.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:53.315+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:53.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:53.315+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:53.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:53.316+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:54.108 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:53.848+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:54.108 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:53.848+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:54.108 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:53.848+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:54.108 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:53.849+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:54.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:54.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:54.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:54.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:54.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:54.107+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:54.267+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:54.267+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:54.267+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:54.268+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:55.223 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:55.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:55.223 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:55.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:55.223 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:55.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:55.223 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:55.081+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:55.224 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:54.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:55.224 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:54.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:55.224 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:54.889+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:55.224 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:54.890+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:55.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:55.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:55.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:55.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:55.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:55.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:55.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:55.222+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:56.261 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:56.119+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:56.261 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:56.119+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:56.262 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:56.119+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:56.262 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:56.120+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:56.262 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:55.888+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:56.262 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:55.888+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:56.262 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:55.888+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:56.262 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:55.889+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:56.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:56.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:56.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:56.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:56.260+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:57.162 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:56.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:57.163 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:56.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:57.163 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:56.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:57.163 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:56.909+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:57.163 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:02:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:02:56.915+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11792 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:02:57.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:57.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:57.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:57.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:57.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:57.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:57.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:57.162+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:57.279+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:57.279+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:57.279+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:57.280+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:58.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:57.934+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:58.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:57.934+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:58.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:57.934+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:58.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:57.935+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:58.313 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:58.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:58.313 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:58.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:58.313 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:58.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:58.314 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:58.182+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:58.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:58.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:58.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:58.312+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:02:59.026 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:58.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:59.026 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:58.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:59.026 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:58.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:02:59.026 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:58.909+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:59.308 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:59.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:59.308 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:59.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:59.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:59.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:02:59.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:02:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:02:59.163+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:02:59.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:59.306+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:59.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:59.306+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:59.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:59.306+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:02:59.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:02:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:02:59.307+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:00.196 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:59.888+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:00.197 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:59.888+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:00.197 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:59.888+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:00.197 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:02:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:02:59.889+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:00.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:00.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:00.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:00.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:00.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:00.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:00.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:00.195+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:00.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:00.342+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:00.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:00.342+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:00.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:00.342+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:00.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:00.343+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:01.189 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:00.885+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:01.189 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:00.885+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:01.189 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:00.885+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:01.189 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:00.886+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:01.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:01.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:01.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:01.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:01.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:01.187+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:01.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:01.295+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:01.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:01.295+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:01.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:01.295+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:01.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:01.296+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:02.184 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:03:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:03:01.915+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11847 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:03:02.184 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:01.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:02.184 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:01.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:02.184 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:01.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:02.184 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:01.908+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:02.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:02.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:02.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:02.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:02.182+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:02.282+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:02.282+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:02.282+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:02.283+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:03.179 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:02.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:03.179 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:02.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:03.179 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:02.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:03.179 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:02.920+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:03.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:03.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:03.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:03.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:03.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:03.177+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:03.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:03.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:03.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:03.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:03.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:03.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:03.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:03.271+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:04.259 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:04.137+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:04.259 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:04.137+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:04.259 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:04.137+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:04.259 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:04.138+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:04.259 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:03.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:04.259 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:03.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:04.259 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:03.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:04.259 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:03.902+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:04.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:04.256+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:04.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:04.256+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:04.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:04.256+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:04.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:04.257+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:05.137 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:04.872+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:05.137 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:04.872+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:05.137 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:04.872+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:05.138 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:04.873+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:05.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:05.136+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:05.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:05.136+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:05.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:05.136+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:05.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:05.137+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:05.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:05.292+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:05.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:05.292+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:05.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:05.292+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:05.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:05.292+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:06.113 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:05.856+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:06.113 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:05.856+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:06.114 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:05.856+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:06.114 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:05.857+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:06.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:06.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:06.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:06.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:06.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:06.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:06.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:06.312+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:06.112+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:06.112+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:06.112+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:06.113+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:07.314 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:03:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:03:06.917+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11904 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:03:07.314 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:06.842+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:07.314 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:06.842+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:07.315 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:06.842+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:07.315 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:06.843+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:07.315 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:07.082+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:07.315 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:07.082+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:07.315 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:07.082+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:07.315 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:07.082+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:07.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:07.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:07.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:07.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:07.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:07.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:07.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:07.313+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:08.257 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:07.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:08.257 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:07.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:08.257 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:07.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:08.257 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:07.883+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:08.258 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:08.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:08.258 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:08.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:08.258 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:08.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:08.258 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:08.120+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:08.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:08.291+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:08.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:08.291+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:08.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:08.291+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:08.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:08.291+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:09.165 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:08.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:09.165 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:08.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:09.165 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:08.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:09.165 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:08.896+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:09.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:09.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:09.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:09.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:09.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:09.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:09.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:09.296+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:09.164+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:09.164+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:09.164+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:09.164+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:10.195 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:09.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:10.196 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:09.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:10.196 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:09.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:10.196 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:09.939+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:10.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:10.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:10.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:10.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:10.195+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:10.319+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:10.319+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:10.319+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:10.320+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:11.184 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:10.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:11.184 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:10.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:11.184 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:10.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:11.184 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:10.928+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:11.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:11.182+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:11.182+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:11.182+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:11.183+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:11.336+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:11.336+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:11.336+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:11.337+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:12.217 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:03:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:03:11.918+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 11961 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:03:12.217 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:11.959+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:12.217 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:11.959+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:12.217 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:11.959+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:12.217 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:11.960+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:12.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:12.215+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:12.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:12.215+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:12.215+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:12.216+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:12.385+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:12.385+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:12.385+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:12.385+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:13.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:13.174+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:13.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:13.174+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:13.174+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:13.175+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:12.958+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:12.958+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:12.958+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:12.959+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:13.393+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:13.393+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:13.393+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:13.394+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:14.205 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:13.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:14.206 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:13.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:14.206 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:13.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:14.206 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:13.912+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:14.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:14.204+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:14.204+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:14.204+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:14.204+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:14.363+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:14.363+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:14.363+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:14.363+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:15.214 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:14.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:15.214 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:14.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:15.214 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:14.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:15.215 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:14.907+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:15.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:15.212+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:15.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:15.212+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:15.212+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:15.213+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:15.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:15.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:15.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:15.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:15.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:15.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:15.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:15.404+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:16.189 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:15.956+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:16.189 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:15.956+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:16.190 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:15.956+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:16.190 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:15.957+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:16.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:16.188+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:16.188+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:16.188+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:16.189+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:16.396+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:16.396+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:16.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:16.396+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:16.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:16.397+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:17.225 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:16.932+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:17.225 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:16.932+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:17.225 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:16.932+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:17.225 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:16.933+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:17.225 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:03:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:03:16.920+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12017 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:03:17.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:17.223+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:17.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:17.223+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:17.223+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:17.224+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:17.390+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:17.390+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:17.390+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:17.391+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:18.215 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:17.923+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:18.215 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:17.923+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:18.215 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:17.923+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:18.215 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:17.924+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:18.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:18.213+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:18.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:18.213+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:18.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:18.213+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:18.214+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:18.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:18.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:18.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:18.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:18.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:18.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:18.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:18.378+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:19.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:19.178+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:19.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:19.178+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:19.178+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:19.178+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:19.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:18.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:19.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:18.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:19.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:18.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:19.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:18.942+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:19.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:19.340+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:19.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:19.340+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:19.340+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:19.341+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:20.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:20.139+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:20.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:20.139+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:20.139+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:20.140+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:19.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:19.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:19.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:20.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:19.951+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:20.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:20.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:20.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:20.331+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:21.189 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:20.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:21.189 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:20.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:21.189 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:20.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:21.189 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:20.931+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:21.190 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:21.188+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:21.190 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:21.188+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:21.190 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:21.188+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:21.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:21.189+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:21.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:21.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:21.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:21.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:21.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:21.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:21.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:21.363+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:22.210 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:03:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:03:21.921+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12074 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:03:22.210 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:21.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:22.210 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:21.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:22.210 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:21.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:22.210 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:21.953+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:22.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:22.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:22.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:22.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:22.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:22.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:22.404+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:22.208+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:22.208+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:22.208+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:22.209+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:23.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:22.995+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:23.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:22.995+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:22.995+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:22.995+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:23.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:23.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:23.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:23.190+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:23.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:23.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:23.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:23.447+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:24.232 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:23.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:24.232 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:23.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:24.232 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:23.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:24.232 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:23.974+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:24.498 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:24.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:24.498 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:24.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:24.499 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:24.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:24.499 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:24.422+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:24.499 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:24.230+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:24.499 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:24.230+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:24.499 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:24.230+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:24.499 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:24.231+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:25.232 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:24.924+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:25.232 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:24.924+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:25.232 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:24.924+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:25.232 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:24.925+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:25.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:25.230+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:25.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:25.230+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:25.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:25.230+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:25.231+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:25.388+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:25.388+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:25.388+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:25.389+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:26.258 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:25.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:26.259 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:25.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:26.259 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:25.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:26.259 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:25.927+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:26.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:26.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:26.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:26.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:26.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:26.257+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:26.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:26.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:26.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:26.411+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:26.957 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:03:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:03:26.922+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12131 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:03:27.226 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:26.956+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:27.226 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:26.956+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:27.226 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:26.956+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:27.226 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:26.957+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:27.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:27.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:27.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:27.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:27.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:27.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:27.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:27.415+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:27.224+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:27.224+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:27.224+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:27.225+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:28.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:28.178+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:28.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:28.178+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:28.178+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:28.179+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:28.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:28.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:28.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:28.003+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:28.409+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:28.409+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:28.409+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:28.410+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:29.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:28.962+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:29.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:28.962+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:29.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:28.962+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:29.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:28.962+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:29.198+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:29.198+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:29.198+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:29.199+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:29.761 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:29.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:29.761 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:29.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:29.761 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:29.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:29.761 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:29.420+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:30.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:30.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:30.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:30.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:30.153+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:29.972+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:29.972+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:29.972+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:30.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:29.973+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:30.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:30.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:30.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:30.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:30.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:30.416+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:30.951+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:30.951+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:30.951+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:30.951+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:31.136+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:31.136+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:31.136+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:31.137+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:31.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:31.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:31.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:31.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:31.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:31.419+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:32.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:03:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:03:31.923+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12188 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:03:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:31.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:31.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:31.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:31.975+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:32.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:32.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:32.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:32.123+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:32.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:32.411+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:32.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:32.411+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:32.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:32.411+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:32.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:32.412+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:33.101+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:33.101+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:33.101+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:00.149222+0000 (oldest deadline 2024-09-10T03:53:20.149222+0000) 2024-09-10T04:03:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:33.101+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:32.949+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:32.949+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:32.949+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:32.950+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:33.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:33.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:33.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:33.423+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:34.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:34.144+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:33.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:33.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:33.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:33.965+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:34.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:34.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:34.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:34.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:34.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:34.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:34.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:34.418+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:35.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:34.944+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:35.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:34.944+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:35.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:34.944+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:35.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:34.945+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:35.436 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:35.165+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:35.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:35.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:35.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:35.435+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:36.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:36.145+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:35.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:35.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:35.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:35.914+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:36.455+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:36.455+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:36.455+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:36.456+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:37.190 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:03:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:03:36.924+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12244 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:03:37.191 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:36.868+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:37.191 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:36.868+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:37.191 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:36.868+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:37.191 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:36.869+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:37.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:37.188+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:37.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:37.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:37.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:37.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:37.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:37.420+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:38.176 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:37.900+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:38.177 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:37.900+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:38.177 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:37.900+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:38.177 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:37.900+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:38.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:38.174+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:38.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:38.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:38.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:38.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:38.413+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:39.134+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:38.909+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:38.909+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:38.909+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:38.909+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:39.804 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:39.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:39.804 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:39.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:39.804 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:39.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:39.804 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:39.447+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:40.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:39.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:40.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:39.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:40.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:39.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:40.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:39.908+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:40.480 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:40.157+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:40.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:40.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:40.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:40.479+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:41.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:41.194+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:40.958+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:40.958+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:40.958+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:40.958+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:41.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:41.449+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:41.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:41.449+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:41.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:41.449+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:41.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:41.449+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:42.204 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:03:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:03:41.925+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12301 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:03:42.204 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:41.970+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:42.204 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:41.970+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:42.204 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:41.970+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:04.631936+0000 (oldest deadline 2024-09-10T03:53:24.631936+0000) 2024-09-10T04:03:42.204 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:41.971+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:42.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:42.202+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:42.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:42.440+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:42.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:42.440+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:42.440+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T03:53:15.127130+0000 (oldest deadline 2024-09-10T03:53:35.127130+0000) 2024-09-10T04:03:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:42.441+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:43.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:43.193+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:43.000+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:43.464+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:44.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:44.201+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:44.031+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:44.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:44.429+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:45.212 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:45.018+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:45.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:45.211+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:45.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:45.401+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:46.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:46.016+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:46.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:46.191+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:46.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:46.357+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:47.232 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:46.976+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:47.232 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:03:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:03:46.927+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12358 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:03:47.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:47.230+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:47.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:47.374+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:48.276 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:48.002+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:48.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:48.275+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:48.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:48.376+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:49.320 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:48.993+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:49.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:49.319+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:49.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:49.348+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:50.296 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:50.000+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:50.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:50.295+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:50.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:50.327+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:51.288 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:51.013+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:51.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:51.287+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:51.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:51.350+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:52.307 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:52.054+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:52.308 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:03:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:03:51.929+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12414 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:03:52.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:52.312+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:52.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:52.306+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:53.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:53.281+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:53.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:53.316+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:53.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:53.079+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:54.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:54.302+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:54.096+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:54.320+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:55.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:55.131+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:55.313 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:55.289+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:55.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:55.312+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:56.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:56.101+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:56.316+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:56.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:56.348+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:03:56.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:56.348+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:03:56.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:56.348+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:03:56.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:56.349+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:57.293 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:03:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:03:56.930+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12470 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:03:57.293 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:57.136+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:57.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:57.291+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:57.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:57.353+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:03:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:57.353+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:03:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:57.353+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:03:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:57.353+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:58.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:58.184+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:58.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:58.323+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:58.310+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:03:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:58.310+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:03:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:58.310+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:03:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:58.311+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:59.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:03:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:03:59.232+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:03:59.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:03:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:03:59.303+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:03:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:59.332+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:03:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:59.332+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:03:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:59.332+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:03:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:03:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:03:59.333+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:00.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:00.218+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:00.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:00.317+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:00.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:00.345+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:00.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:00.345+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:00.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:00.345+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:00.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:00.345+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:01.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:01.182+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:01.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:01.364+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:01.311+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:01.311+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:01.311+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:01.312+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:02.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:04:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:04:01.931+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12527 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:04:02.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:02.376+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:02.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:02.221+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:02.311+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:02.311+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:02.311+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:02.312+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:03.553 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:03.193+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:03.553 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:03.373+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:03.554 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:03.265+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:03.554 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:03.265+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:03.554 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:03.265+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:03.554 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:03.266+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:04.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:04.192+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:04.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:04.423+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:04.220+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:04.220+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:04.220+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:04.221+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:05.447 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:05.180+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:05.447 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:05.205+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:05.447 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:05.205+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:05.448 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:05.205+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:05.448 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:05.206+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:05.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:05.446+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:06.434 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:06.165+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:06.435 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:06.165+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:06.435 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:06.165+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:06.435 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:06.165+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:06.435 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:06.186+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:06.435 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:06.186+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:06.435 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:06.186+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:06.435 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:06.187+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:06.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:06.433+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:06.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:06.433+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:06.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:06.433+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:06.434+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:07.206 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:04:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:04:06.932+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12583 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:04:07.207 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:07.181+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:07.207 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:07.181+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:07.207 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:07.181+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:07.207 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:07.182+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:07.461 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:07.205+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:07.461 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:07.205+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:07.462 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:07.205+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:07.462 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:07.206+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:07.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:07.460+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:07.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:07.460+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:07.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:07.460+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:07.460+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:08.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:08.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:08.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:08.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:08.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:08.204+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:08.237+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:08.237+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:08.237+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:08.238+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:08.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:08.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:08.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:08.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:08.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:08.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:08.416+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:09.445 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:09.191+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:09.445 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:09.191+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:09.445 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:09.191+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:09.445 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:09.192+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:09.445 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:09.235+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:09.446 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:09.235+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:09.446 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:09.235+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:09.446 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:09.235+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:09.704 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:09.443+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:09.704 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:09.443+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:09.704 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:09.443+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:09.704 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:09.444+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:10.482 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:10.235+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:10.482 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:10.235+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:10.482 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:10.235+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:10.483 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:10.235+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:10.483 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:10.192+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:10.483 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:10.192+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:10.483 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:10.192+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:10.483 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:10.193+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:10.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:10.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:10.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:10.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:10.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:10.481+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:11.505 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:11.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:11.505 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:11.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:11.505 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:11.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:11.505 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:11.229+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:11.505 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:11.219+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:11.506 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:11.219+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:11.506 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:11.219+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:11.506 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:11.220+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:11.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:11.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:11.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:11.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:11.504+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:12.186 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:04:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:04:11.934+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12639 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:04:12.474 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:12.198+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:12.474 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:12.198+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:12.475 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:12.198+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:12.475 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:12.198+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:12.475 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:12.185+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:12.475 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:12.185+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:12.475 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:12.185+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:12.475 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:12.186+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:12.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:12.473+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:12.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:12.473+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:12.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:12.473+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:12.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:12.474+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:13.497 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:13.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:13.497 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:13.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:13.497 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:13.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:13.498 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:13.177+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:13.498 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:13.186+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:13.498 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:13.186+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:13.498 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:13.186+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:13.498 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:13.187+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:13.494+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:13.494+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:13.494+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:13.495+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:14.452 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:14.171+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:14.452 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:14.171+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:14.452 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:14.171+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:14.452 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:14.172+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:14.452 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:14.185+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:14.453 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:14.185+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:14.453 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:14.185+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:14.453 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:14.186+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:14.449+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:14.449+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:14.449+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:14.450+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:15.425 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:15.182+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:15.425 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:15.182+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:15.425 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:15.182+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:15.425 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:15.183+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:15.425 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:15.153+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:15.425 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:15.153+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:15.426 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:15.153+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:15.426 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:15.154+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:15.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:15.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:15.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:15.423+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:16.436 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:16.105+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:16.436 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:16.105+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:16.436 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:16.105+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:16.436 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:16.106+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:16.437 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:16.135+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:16.437 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:16.135+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:16.437 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:16.135+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:16.437 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:16.136+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:16.433+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:16.433+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:16.433+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:16.434+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:17.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:04:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:04:16.934+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12694 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:04:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:17.060+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:17.060+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:17.060+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:17.061+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:17.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:17.113+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:17.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:17.113+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:17.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:17.113+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:17.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:17.114+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:17.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:17.388+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:17.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:17.388+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:17.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:17.388+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:17.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:17.389+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:18.413 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:18.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:18.413 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:18.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:18.413 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:18.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:18.414 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:18.089+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:18.414 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:18.123+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:18.414 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:18.123+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:18.414 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:18.123+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:18.414 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:18.124+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:18.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:18.411+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:18.411+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:18.411+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:18.411+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:19.166 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:19.137+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:19.166 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:19.137+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:19.166 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:19.137+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:19.166 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:19.138+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:19.432 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:19.164+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:19.432 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:19.164+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:19.432 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:19.164+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:19.432 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:19.164+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:19.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:19.430+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:19.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:19.430+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:19.430+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:19.431+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:20.404 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:20.108+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:20.405 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:20.108+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:20.405 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:20.108+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:20.405 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:20.109+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:20.405 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:20.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:20.405 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:20.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:20.405 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:20.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:20.405 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:20.205+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:20.400+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:20.400+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:20.400+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:20.401+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:21.442 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:21.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:21.443 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:21.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:21.443 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:21.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:21.443 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:21.224+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:21.443 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:21.156+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:21.443 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:21.156+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:21.443 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:21.156+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:21.443 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:21.157+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:21.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:21.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:21.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:21.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:21.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:21.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:21.442+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:22.227 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:04:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:04:21.936+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12753 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:04:22.227 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:22.225+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:22.227 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:22.225+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:22.227 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:22.225+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:22.228 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:22.114+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:22.228 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:22.114+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:22.228 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:22.114+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:22.228 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:22.115+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:22.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:22.226+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:22.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:22.414+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:22.414+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:22.414+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:22.415+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:23.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:23.075+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:23.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:23.075+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:23.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:23.075+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:23.076+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:23.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:23.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:23.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:23.204+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:23.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:23.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:23.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:23.442+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:24.485 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:24.095+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:24.485 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:24.095+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:24.485 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:24.095+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:24.485 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:24.096+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:24.486 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:24.180+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:24.486 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:24.180+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:24.486 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:24.180+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:24.486 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:24.181+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:24.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:24.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:24.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:24.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:24.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:24.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:24.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:24.484+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:25.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:25.157+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:25.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:25.157+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:25.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:25.157+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:25.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:25.158+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:25.077+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:25.077+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:25.077+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:25.078+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:25.465+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:25.465+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:25.465+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:25.465+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:26.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:26.073+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:26.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:26.073+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:26.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:26.073+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:26.074+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:26.174+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:26.174+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:26.174+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:26.175+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:26.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:26.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:26.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:26.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:26.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:26.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:26.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:26.461+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:27.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:27.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:27.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:27.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:27.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:27.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:27.064+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:27.133+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:27.133+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:27.133+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:27.134+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:27.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:04:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:04:26.938+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12812 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:04:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:27.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:27.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:27.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:27.447+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:28.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:28.069+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:28.069+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:28.069+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:28.070+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:28.092+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:28.092+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:28.092+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:28.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:28.093+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:28.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:28.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:28.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:28.494+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:29.454 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:29.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:29.454 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:29.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:29.454 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:29.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:29.455 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:29.088+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:29.455 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:29.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:29.455 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:29.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:29.455 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:29.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:29.455 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:29.097+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:29.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:29.453+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:29.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:29.453+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:29.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:29.453+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:29.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:29.453+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:30.460 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:30.113+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:30.460 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:30.113+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:30.461 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:30.113+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:30.461 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:30.114+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:30.461 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:30.107+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:30.461 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:30.107+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:30.461 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:30.107+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:30.461 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:30.107+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:30.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:30.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:30.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:30.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:30.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:30.460+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:31.483 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:31.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:31.483 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:31.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:31.483 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:31.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:31.483 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:31.096+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:31.484 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:31.107+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:31.484 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:31.107+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:31.484 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:31.107+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:31.484 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:31.108+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:31.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:31.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:31.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:31.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:31.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:31.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:31.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:31.482+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:32.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:32.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:32.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:32.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:32.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:32.125+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:32.133+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:32.133+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:32.133+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:32.134+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:32.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:04:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:04:31.939+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12871 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:04:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:32.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:32.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:32.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:32.486+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:33.501 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:33.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:33.502 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:33.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:33.502 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:33.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:33.502 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:33.170+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:33.502 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:33.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:33.502 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:33.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:33.502 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:33.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:33.502 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:33.124+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:33.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:33.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:33.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:33.501+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:34.483 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:34.115+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:34.483 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:34.115+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:34.483 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:34.115+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:34.484 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:34.116+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:34.484 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:34.161+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:34.484 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:34.161+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:34.484 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:34.161+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:34.484 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:34.162+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:34.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:34.481+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:34.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:34.481+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:34.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:34.481+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:34.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:34.481+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:35.506 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:35.114+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:35.506 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:35.114+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:35.506 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:35.114+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:35.506 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:35.115+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:35.506 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:35.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:35.506 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:35.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:35.506 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:35.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:35.506 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:35.084+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:35.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:35.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:35.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:35.505+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:36.525 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:36.142+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:36.526 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:36.142+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:36.526 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:36.142+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:36.526 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:36.143+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:36.526 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:36.105+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:36.526 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:36.105+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:36.526 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:36.105+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:36.526 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:36.106+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:36.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:36.524+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:36.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:36.524+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:36.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:36.524+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:36.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:36.525+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:37.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:04:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:04:36.941+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12929 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:04:37.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:37.069+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:37.069+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:37.069+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:37.069+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:37.171+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:37.171+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:37.171+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:37.172+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:37.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:37.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:37.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:37.494+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:38.533 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:38.085+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:38.533 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:38.085+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:38.533 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:38.085+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:38.533 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:38.086+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:38.533 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:38.198+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:38.534 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:38.198+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:38.534 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:38.198+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:38.534 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:38.199+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:38.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:38.531+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:38.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:38.531+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:38.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:38.531+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:38.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:38.532+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:39.199+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:39.199+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:39.199+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:39.200+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:39.069+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:39.069+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:39.069+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:39.070+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:39.488+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:39.488+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:39.488+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:39.489+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:40.178+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:40.178+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:40.178+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:40.179+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:40.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:40.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:40.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:40.030+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:40.821 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:40.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:40.821 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:40.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:40.821 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:40.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:40.821 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:40.495+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:41.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:41.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:41.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:41.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:41.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:41.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:41.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:41.029+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:41.479 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:41.212+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:41.480 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:41.212+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:41.480 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:41.212+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:41.480 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:41.213+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:41.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:41.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:41.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:41.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:41.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:41.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:41.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:41.479+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:42.257 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:04:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:04:41.942+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 12988 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:04:42.257 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:42.056+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:42.258 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:42.056+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:42.258 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:42.056+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:42.258 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:42.057+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:42.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:42.256+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:42.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:42.256+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:42.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:42.256+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:42.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:42.256+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:42.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:42.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:42.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:42.487+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:43.053+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:43.053+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:43.053+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:43.054+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:43.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:43.295+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:43.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:43.295+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:43.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:43.295+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:43.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:43.295+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:43.475+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:43.475+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:43.475+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:43.476+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:44.265+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:44.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:44.265+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:44.265+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:44.266+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:44.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:44.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:44.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:44.035+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:44.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:44.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:44.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:44.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:44.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:44.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:44.490+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:45.019+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:45.019+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:45.019+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:45.019+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:45.267+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:45.267+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:45.267+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:45.268+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:45.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:45.511+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:45.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:45.511+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:45.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:45.511+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:45.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:45.511+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:46.266 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:46.050+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:46.266 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:46.050+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:46.267 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:46.050+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:46.267 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:46.050+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:46.537 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:46.265+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:46.537 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:46.265+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:46.537 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:46.265+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:46.537 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:46.266+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:46.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:46.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:46.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:46.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:46.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:46.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:46.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:46.536+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:47.248 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:04:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:04:46.944+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 13046 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:04:47.248 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:47.047+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:47.248 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:47.047+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:47.248 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:47.047+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:47.248 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:47.048+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:47.559 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:47.244+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:47.559 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:47.244+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:47.560 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:47.244+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:47.560 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:47.245+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:47.558+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:47.558+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:47.558+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:47.559+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:48.227 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:48.070+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:48.228 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:48.070+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:48.228 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:48.070+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:48.228 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:48.070+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:48.523 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:48.226+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:48.523 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:48.226+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:48.523 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:48.226+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:48.523 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:48.227+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:48.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:48.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:48.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:48.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:48.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:48.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:48.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:48.523+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:49.477 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:49.096+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:49.477 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:49.096+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:49.477 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:49.096+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:49.478 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:49.097+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:49.478 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:49.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:49.478 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:49.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:49.478 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:49.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:49.478 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:49.274+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:49.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:49.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:49.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:49.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:49.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:49.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:49.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:49.476+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:50.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:50.090+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:50.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:50.090+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:50.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:50.090+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:50.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:50.091+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:50.481 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:50.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:50.481 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:50.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:50.481 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:50.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:50.482 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:50.316+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:50.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:50.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:50.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:50.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:50.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:50.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:50.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:50.479+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:51.308 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:51.114+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:51.308 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:51.114+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:51.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:51.114+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:51.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:51.114+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:51.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:51.484+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:51.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:51.484+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:51.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:51.484+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:51.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:51.485+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:51.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:51.306+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:51.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:51.306+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:51.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:51.306+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:51.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:51.307+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:52.324 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:04:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:04:51.944+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 13105 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:04:52.324 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:52.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:52.324 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:52.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:52.324 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:52.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:52.324 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:52.149+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:52.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:52.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:52.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:52.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:52.454+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:52.321+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:52.321+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:52.321+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:52.322+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:53.507 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:53.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:53.508 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:53.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:53.508 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:53.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:53.508 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:53.164+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:53.508 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:53.295+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:53.508 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:53.295+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:53.508 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:53.295+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:53.508 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:53.296+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:53.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:53.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:53.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:53.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:53.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:53.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:53.506+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:54.487 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:54.179+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:54.487 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:54.179+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:54.488 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:54.179+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:54.488 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:54.180+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:54.488 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:54.314+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:54.488 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:54.314+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:54.488 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:54.314+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:54.488 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:54.315+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:54.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:54.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:54.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:54.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:54.486+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:55.509 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:55.177+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:55.509 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:55.177+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:55.509 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:55.177+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:55.509 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:55.178+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:55.509 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:55.355+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:55.509 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:55.355+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:55.509 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:55.355+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:55.510 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:55.356+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:55.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:55.506+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:55.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:55.506+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:55.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:55.506+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:55.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:55.507+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:56.383 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:56.184+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:56.383 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:56.184+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:56.383 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:56.184+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:56.383 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:56.185+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:56.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:56.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:56.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:56.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:56.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:56.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:56.381+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:56.484+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:56.484+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:56.484+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:56.484+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:57.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:04:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:04:56.946+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 13163 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:04:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:57.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:57.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:57.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:57.151+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:57.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:57.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:57.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:57.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:57.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:57.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:57.448+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:57.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:57.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:57.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:57.354+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:58.407 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:58.154+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:58.407 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:58.154+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:58.407 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:58.154+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:58.408 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:58.155+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:58.408 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:58.375+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:58.408 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:58.375+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:58.408 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:58.375+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:58.408 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:58.376+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:58.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:58.406+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:58.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:58.406+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:58.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:58.406+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:58.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:58.407+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:04:59.411 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:59.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:59.411 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:59.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:59.412 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:59.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:04:59.412 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:04:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:04:59.148+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:59.412 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:59.326+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:59.412 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:59.326+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:59.412 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:59.326+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:04:59.412 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:04:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:04:59.327+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:04:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:59.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:59.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:59.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:04:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:04:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:04:59.411+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:00.447 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:00.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:00.447 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:00.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:00.448 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:00.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:00.448 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:00.307+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:00.448 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:00.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:00.448 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:00.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:00.448 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:00.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:00.448 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:00.148+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:00.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:00.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:00.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:00.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:00.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:00.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:00.447+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:01.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:01.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:01.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:01.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:01.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:01.164+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:01.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:01.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:01.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:01.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:01.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:01.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:01.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:01.405+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:01.348+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:01.348+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:01.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:01.348+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:01.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:01.349+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:02.214 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:05:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:05:01.948+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 13222 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:05:02.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:02.213+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:02.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:02.213+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:02.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:02.213+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:02.213+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:02.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:02.354+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:02.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:02.354+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:02.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:02.354+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:02.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:02.355+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:02.363+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:02.363+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:02.363+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:02.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:02.363+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:03.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:03.204+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:03.204+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:03.204+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:03.205+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:03.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:03.376+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:03.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:03.376+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:03.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:03.376+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:03.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:03.377+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:03.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:03.405+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:03.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:03.405+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:03.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:03.405+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:03.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:03.405+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:04.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:04.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:04.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:04.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:04.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:04.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:04.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:04.401+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:04.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:04.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:04.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:04.167+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:04.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:04.408+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:04.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:04.408+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:04.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:04.408+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:04.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:04.409+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:05.216+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:05.216+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:05.216+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:05.216+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:05.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:05.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:05.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:05.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:05.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:05.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:05.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:05.379+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:05.425+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:05.425+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:05.425+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:05.426+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:06.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:06.386+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:06.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:06.386+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:06.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:06.386+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:06.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:06.387+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:06.253+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:06.253+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:06.253+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:06.253+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:06.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:06.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:06.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:06.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:06.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:06.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:06.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:06.384+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:07.282 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:05:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:05:06.949+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 13281 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:05:07.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:07.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:07.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:07.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:07.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:07.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:07.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:07.381+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:07.360+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:07.360+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:07.360+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:07.360+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:07.280+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:07.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:07.280+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:07.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:07.280+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:07.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:07.281+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:08.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:08.255+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:08.255+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:08.255+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:08.255+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:08.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:08.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:08.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:08.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:08.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:08.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:08.375+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:08.346+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:08.346+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:08.346+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:08.347+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:09.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:09.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:09.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:09.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:09.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:09.401+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:09.238+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:09.238+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:09.238+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:09.239+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:09.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:09.388+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:09.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:09.388+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:09.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:09.388+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:09.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:09.389+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:10.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:10.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:10.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:10.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:10.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:10.258+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:10.389+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:10.389+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:10.389+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:10.390+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:10.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:10.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:10.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:10.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:10.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:10.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:10.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:10.437+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:11.391 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:11.286+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:11.391 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:11.286+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:11.391 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:11.286+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:11.391 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:11.286+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:11.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:11.426+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:11.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:11.426+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:11.426+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:11.427+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:11.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:11.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:11.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:11.441+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:12.241 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:05:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:05:11.950+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 13340 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:05:12.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:12.240+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:12.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:12.240+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:12.240+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:12.240+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:12.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:12.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:12.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:12.448+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:12.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:12.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:12.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:12.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:12.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:12.395+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:13.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:13.217+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:13.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:13.217+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:13.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:13.217+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:13.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:13.218+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:13.460+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:13.460+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:13.460+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:13.461+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:13.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:13.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:13.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:13.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:13.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:13.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:13.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:13.411+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:14.493 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:14.251+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:14.493 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:14.251+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:14.493 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:14.251+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:14.493 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:14.252+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:14.494 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:14.448+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:14.494 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:14.448+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:14.494 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:14.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:14.494 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:14.450+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:14.490+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:14.490+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:14.490+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:14.491+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:15.467 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:15.215+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:15.467 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:15.215+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:15.467 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:15.215+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:15.468 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:15.216+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:15.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:15.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:15.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:15.500+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:15.466+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:15.466+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:15.466+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:15.466+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:16.460 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:16.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:16.460 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:16.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:16.461 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:16.209+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:16.461 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:16.209+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:16.461 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:16.209+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:16.461 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:16.210+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:16.461 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:16.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:16.461 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:16.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:16.462 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:16.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:16.462 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:16.450+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:16.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:16.459+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:17.254 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:05:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:05:16.952+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 13398 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:05:17.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:17.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:17.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:17.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:17.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:17.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:17.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:17.414+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:17.252+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:17.252+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:17.252+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:17.253+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:17.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:17.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:17.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:17.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:17.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:17.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:17.470+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:18.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:18.269+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:18.269+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:18.269+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:18.270+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:18.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:18.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:18.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:18.413+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:18.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:18.498+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:18.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:18.498+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:18.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:18.498+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:18.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:18.499+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:19.516 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:19.262+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:19.516 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:19.262+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:19.516 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:19.262+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:19.516 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:19.263+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:19.516 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:19.457+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:19.516 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:19.457+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:19.516 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:19.457+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:19.517 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:19.458+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:19.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:19.514+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:19.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:19.514+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:19.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:19.514+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:19.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:19.515+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:20.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:20.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:20.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:20.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:20.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:20.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:20.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:20.483+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:20.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:20.247+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:20.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:20.247+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:20.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:20.247+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:20.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:20.247+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:20.472+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:20.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:20.472+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:20.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:20.472+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:20.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:20.472+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:21.568 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:21.255+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:21.568 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:21.255+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:21.568 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:21.255+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:21.568 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:21.255+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:21.568 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:21.452+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:21.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:21.452+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:21.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:21.452+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:21.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:21.453+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:21.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:21.482+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:21.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:21.482+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:21.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:21.482+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:21.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:21.483+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:22.261 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:05:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:05:21.953+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 13457 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:05:22.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:22.486+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:22.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:22.486+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:22.486+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:22.487+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:22.259+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:22.259+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:22.259+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:22.260+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:22.494+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:22.494+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:22.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:22.494+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:22.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:22.495+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:23.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:23.483+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:23.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:23.483+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:23.483+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:23.484+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:23.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:23.300+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:23.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:23.300+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:23.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:23.300+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:23.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:23.301+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:23.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:23.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:23.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:23.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:23.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:23.508+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:24.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:24.504+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:24.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:24.504+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:24.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:24.504+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:24.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:24.505+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:24.598 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:24.466+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:24.599 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:24.466+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:24.599 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:24.466+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:24.599 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:24.467+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:24.599 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:24.333+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:24.599 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:24.333+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:24.599 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:24.333+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:24.599 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:24.334+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:25.347+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:25.347+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:25.347+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:25.347+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:25.457+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:25.457+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:25.457+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:25.458+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:25.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:25.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:25.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:25.484+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:26.649 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:26.349+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:26.649 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:26.349+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:26.650 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:26.349+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:26.650 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:26.350+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:26.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:26.520+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:26.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:26.520+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:26.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:26.520+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:26.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:26.521+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:26.651 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:26.408+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:26.651 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:26.408+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:26.651 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:26.408+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:26.651 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:26.409+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:27.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:05:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:05:26.955+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 13516 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:05:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:27.471+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:27.471+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:27.471+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:27.472+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:27.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:27.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:27.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:27.413+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:27.385+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:27.385+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:27.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:27.385+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:27.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:27.386+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:28.369+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:28.369+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:28.369+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:28.370+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:28.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:28.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:28.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:28.417+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:28.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:28.453+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:28.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:28.453+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:28.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:28.453+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:28.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:28.454+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:29.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:29.357+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:29.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:29.357+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:29.357+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:29.358+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:29.428+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:29.428+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:29.428+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:29.428+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:29.414+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:29.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:29.414+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:29.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:29.414+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:29.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:29.415+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:30.323+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:30.323+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:30.323+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:30.323+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:30.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:30.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:30.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:30.384+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:30.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:30.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:30.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:30.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:30.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:30.440+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:31.741 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:31.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:31.741 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:31.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:31.742 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:31.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:31.742 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:31.376+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:31.742 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:31.372+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:31.742 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:31.372+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:31.742 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:31.372+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:31.742 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:31.373+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:31.742 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:31.442+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:31.742 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:31.442+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:31.743 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:31.442+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:31.743 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:31.443+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:32.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:05:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:05:31.955+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 13575 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:05:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:32.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:32.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:32.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:32.404+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:32.395+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:32.395+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:32.395+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:32.395+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:32.434+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:32.434+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:32.434+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:32.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:32.435+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:33.359+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:33.359+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:33.359+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:33.359+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:33.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:33.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:33.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:33.407+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:33.390+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:33.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:33.390+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:33.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:33.390+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:33.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:33.390+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:34.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:34.310+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:34.310+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:34.310+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:34.311+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:34.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:34.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:34.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:34.414+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:34.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:34.444+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:34.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:34.444+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:34.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:34.444+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:34.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:34.445+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:35.712 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:35.376+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:35.712 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:35.376+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:35.712 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:35.376+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:35.713 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:35.377+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:35.713 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:35.457+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:35.713 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:35.457+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:35.713 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:35.457+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:35.713 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:35.458+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:35.713 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:35.348+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:35.713 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:35.348+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:35.713 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:35.348+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:35.714 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:35.348+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:36.827 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:36.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:36.828 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:36.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:36.828 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:36.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:36.828 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:36.390+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:36.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:36.488+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:36.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:36.488+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:36.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:36.488+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:36.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:36.488+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:36.829 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:36.340+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:36.829 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:36.340+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:36.829 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:36.340+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:36.829 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:36.341+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:37.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:05:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:05:36.958+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 13630 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:05:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:37.437+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:37.437+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:37.437+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:37.438+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:37.349+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:37.349+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:37.349+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:37.349+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:37.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:37.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:37.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:37.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:37.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:37.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:37.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:37.528+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:38.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:38.385+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:38.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:38.385+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:38.385+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:38.386+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:38.528+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:38.528+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:38.528+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:38.529+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:38.442+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:38.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:38.442+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:38.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:38.442+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:38.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:38.443+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:39.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:39.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:39.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:39.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:39.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:39.407+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:39.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:39.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:39.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:39.378+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:39.518+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:39.518+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:39.518+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:39.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:39.519+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:40.380+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:40.380+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:40.380+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:40.381+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:40.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:40.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:40.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:40.480+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:40.409+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:40.409+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:40.409+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:40.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:40.410+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:41.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:41.444+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:41.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:41.444+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:41.444+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:41.444+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:41.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:41.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:41.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:41.510+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:41.430+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:41.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:41.430+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:41.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:41.430+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:41.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:41.430+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:42.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:05:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:05:41.959+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 13686 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:05:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:42.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:42.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:42.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:42.480+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:42.438+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:42.438+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:42.438+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:42.439+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:42.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:42.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:42.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:42.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:42.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:42.558+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:43.455+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:43.455+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:43.455+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:43.456+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:43.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:43.430+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:43.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:43.430+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:43.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:43.430+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:43.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:43.431+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:43.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:43.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:43.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:43.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:43.603+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:44.438+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:44.438+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:44.438+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:44.439+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:44.405+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:44.405+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:44.405+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:44.406+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:44.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:44.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:44.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:44.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:44.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:44.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:44.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:44.569+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:45.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:45.365+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:45.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:45.365+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:45.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:45.365+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:45.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:45.366+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:45.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:45.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:45.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:45.533+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:45.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:45.469+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:45.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:45.469+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:45.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:45.469+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:45.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:45.470+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:46.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:46.516+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:46.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:46.516+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:46.516+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:46.517+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:46.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:46.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:46.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:46.381+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:46.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:46.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:46.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:46.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:46.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:46.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:46.505+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:47.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:05:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:05:46.960+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 13736 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:05:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:47.491+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:47.491+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:47.491+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:47.492+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:47.512+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:47.512+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:47.512+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:47.512+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:47.349+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:47.349+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:47.349+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:47.350+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:48.518+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:48.518+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:48.518+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:48.519+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:48.530+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:48.530+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:48.530+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:48.531+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:48.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:48.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:48.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:48.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:48.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:48.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:48.374+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:49.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:49.481+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:49.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:49.481+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:49.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:49.481+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:49.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:49.482+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:49.730 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:49.521+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:49.730 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:49.521+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:49.730 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:49.521+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:49.730 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:49.522+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:49.730 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:49.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:49.730 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:49.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:49.730 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:49.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:49.730 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:49.420+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:50.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:50.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:50.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:50.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:50.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:50.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:50.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:50.412+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:50.447+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:50.447+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:50.447+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:50.447+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:50.500+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:50.500+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:50.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:50.500+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:50.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:50.500+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:51.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:51.475+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:51.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:51.475+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:51.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:51.475+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:51.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:51.476+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:51.429+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:51.429+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:51.429+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:51.430+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:51.387+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:51.387+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:51.387+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:51.388+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:52.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:05:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:05:51.962+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 13786 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:05:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:52.490+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:52.490+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:52.490+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:52.491+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:52.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:52.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:52.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:52.400+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:52.381+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:52.381+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:52.381+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:52.382+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:53.606 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:53.376+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:53.606 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:53.376+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:53.606 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:53.376+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:53.607 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:53.377+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:53.607 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:53.474+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:53.607 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:53.474+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:53.607 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:53.474+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:53.608 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:53.474+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:53.608 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:53.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:53.609 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:53.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:53.609 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:53.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:53.609 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:53.342+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:54.402 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:54.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:54.402 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:54.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:54.402 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:54.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:54.402 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:54.358+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:54.489+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:54.489+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:54.489+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:54.490+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:54.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:54.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:54.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:54.401+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:55.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:55.333+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:55.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:55.333+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:55.333+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:55.333+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:55.456+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:55.456+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:55.456+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:55.457+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:55.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:55.444+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:55.444+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:55.444+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:55.444+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:56.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:56.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:56.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:56.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:56.358+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:56.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:56.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:56.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:56.488+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:56.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:56.450+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:56.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:56.450+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:56.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:56.450+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:56.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:56.450+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:57.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:05:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:05:56.963+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 13838 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:05:57.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:57.538+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:57.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:57.538+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:57.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:57.538+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:57.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:57.538+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:57.403+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:57.403+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:57.403+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:57.404+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:57.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:57.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:57.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:57.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:57.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:57.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:57.358+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:58.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:58.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:58.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:58.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:58.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:58.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:58.440+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:58.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:58.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:58.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:58.551+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:58.365+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:58.365+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:58.365+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:58.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:58.366+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:05:59.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:59.391+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:59.391+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:59.391+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:05:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:05:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:05:59.391+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:59.583+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:59.583+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:59.583+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:05:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:05:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:05:59.583+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:05:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:59.338+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:59.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:59.338+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:59.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:59.338+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:05:59.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:05:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:05:59.338+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:00.606 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:00.428+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:00.606 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:00.428+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:00.606 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:00.428+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:00.606 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:00.429+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:00.607 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:00.332+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:00.607 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:00.332+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:00.607 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:00.332+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:00.607 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:00.333+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:00.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:00.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:00.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:00.605+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:01.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:01.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:01.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:01.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:01.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:01.384+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:01.381+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:01.381+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:01.381+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:01.382+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:01.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:01.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:01.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:01.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:01.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:01.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:01.627+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:02.274 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:06:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:06:01.965+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 13890 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:06:02.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:02.373+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:02.373+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:02.373+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:02.374+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:02.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:02.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:02.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:02.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:02.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:02.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:02.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:02.425+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:02.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:02.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:02.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:02.649+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:03.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:03.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:03.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:03.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:03.683 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:03.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:03.683 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:03.421+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:03.683 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:03.382+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:03.683 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:03.382+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:03.683 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:03.382+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:03.683 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:03.383+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:04.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:03.681+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:03.681+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:03.681+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:03.681+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:04.690 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:04.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:04.690 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:04.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:04.690 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:04.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:04.690 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:04.413+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:04.691 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:04.374+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:04.691 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:04.374+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:04.691 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:04.374+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:04.691 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:04.375+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:05.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:04.689+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:05.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:04.689+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:04.689+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:04.689+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:05.710 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:05.369+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:05.710 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:05.369+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:05.710 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:05.369+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:05.710 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:05.369+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:05.711 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:05.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:05.711 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:05.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:05.711 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:05.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:05.711 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:05.407+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:06.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:05.708+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:06.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:05.708+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:06.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:05.708+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:06.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:05.709+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:06.742 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:06.349+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:06.742 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:06.349+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:06.742 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:06.349+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:06.742 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:06.350+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:06.742 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:06.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:06.742 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:06.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:06.742 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:06.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:06.743 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:06.438+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:07.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:06:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:06:06.965+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 13949 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:06:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:06.739+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:06.739+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:06.739+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:07.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:06.740+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:07.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:07.427+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:07.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:07.427+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:07.728 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:07.427+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:07.728 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:07.427+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:07.728 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:07.386+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:07.728 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:07.386+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:07.729 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:07.386+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:07.729 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:07.387+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:07.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:07.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:07.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:07.725+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:08.764 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:08.405+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:08.764 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:08.405+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:08.765 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:08.405+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:08.765 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:08.405+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:08.765 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:08.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:08.765 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:08.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:08.765 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:08.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:08.765 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:08.371+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:09.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:08.762+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:09.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:08.762+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:09.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:08.762+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:09.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:08.762+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:09.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:09.407+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:09.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:09.407+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:09.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:09.407+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:09.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:09.407+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:09.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:09.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:09.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:09.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:09.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:09.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:09.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:09.418+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:09.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:09.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:09.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:09.731+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:10.718 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:10.391+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:10.719 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:10.391+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:10.719 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:10.391+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:10.719 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:10.392+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:10.719 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:10.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:10.719 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:10.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:10.719 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:10.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:10.719 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:10.449+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:11.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:10.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:11.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:10.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:10.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:10.717+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:11.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:11.669+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:11.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:11.669+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:11.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:11.669+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:11.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:11.669+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:11.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:11.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:11.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:11.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:11.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:11.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:11.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:11.425+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:11.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:11.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:11.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:11.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:11.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:11.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:11.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:11.500+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:12.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:06:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:06:11.967+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 14007 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:06:12.713 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:12.405+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:12.713 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:12.405+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:12.713 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:12.405+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:12.713 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:12.406+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:12.713 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:12.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:12.714 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:12.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:12.714 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:12.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:12.714 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:12.507+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:12.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:12.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:12.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:12.713+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:13.686 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:13.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:13.686 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:13.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:13.686 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:13.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:13.686 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:13.503+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:13.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:13.417+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:13.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:13.417+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:13.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:13.417+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:13.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:13.418+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:14.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:13.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:14.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:13.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:14.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:13.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:13.685+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:14.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:14.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:14.731 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:14.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:14.731 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:14.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:14.731 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:14.533+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:14.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:14.452+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:14.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:14.452+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:14.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:14.452+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:14.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:14.453+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:15.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:14.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:14.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:14.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:14.730+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:15.730 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:15.472+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:15.730 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:15.472+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:15.730 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:15.472+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:15.730 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:15.473+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:15.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:15.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:15.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:15.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:15.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:15.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:15.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:15.548+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:16.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:15.728+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:16.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:15.728+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:16.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:15.728+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:15.729+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:16.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:16.505+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:16.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:16.505+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:16.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:16.505+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:16.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:16.506+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:16.776 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:16.470+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:16.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:16.470+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:16.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:16.470+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:16.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:16.471+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:17.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:06:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:06:16.969+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 14065 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:06:17.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:16.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:16.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:16.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:16.775+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:17.735 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:17.474+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:17.735 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:17.474+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:17.735 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:17.474+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:17.735 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:17.475+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:17.736 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:17.453+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:17.736 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:17.453+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:17.736 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:17.453+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:17.736 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:17.454+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:18.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:17.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:18.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:17.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:18.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:17.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:18.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:17.734+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:18.721 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:18.438+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:18.721 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:18.438+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:18.721 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:18.438+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:18.721 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:18.438+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:18.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:18.447+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:18.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:18.447+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:18.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:18.447+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:18.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:18.447+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:18.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:18.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:18.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:18.739+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:19.767 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:19.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:19.767 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:19.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:19.767 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:19.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:19.767 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:19.460+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:19.767 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:19.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:19.767 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:19.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:19.767 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:19.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:19.767 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:19.424+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:19.765+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:19.765+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:19.765+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:19.766+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:20.810 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:20.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:20.810 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:20.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:20.810 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:20.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:20.810 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:20.436+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:20.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:20.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:20.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:20.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:20.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:20.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:20.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:20.379+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:21.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:20.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:21.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:20.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:20.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:20.809+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:21.806 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:21.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:21.807 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:21.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:21.807 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:21.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:21.807 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:21.478+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:21.807 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:21.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:21.807 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:21.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:21.807 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:21.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:21.807 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:21.381+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:22.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:21.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:21.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:21.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:21.806+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:22.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:06:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:06:21.970+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 14124 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:06:22.758 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:22.428+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:22.758 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:22.428+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:22.758 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:22.428+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:22.758 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:22.428+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:22.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:22.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:22.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:22.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:22.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:22.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:22.759 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:22.524+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:22.756+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:22.756+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:22.756+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:22.757+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:23.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:23.564+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:23.731 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:23.565+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:23.731 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:23.565+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:23.731 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:23.565+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:23.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:23.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:23.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:23.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:23.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:23.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:23.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:23.404+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:24.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:23.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:24.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:23.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:24.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:23.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:23.730+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:24.772 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:24.385+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:24.772 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:24.385+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:24.772 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:24.385+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:24.773 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:24.386+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:24.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:24.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:24.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:24.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:24.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:24.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:24.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:24.526+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:25.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:24.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:25.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:24.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:25.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:24.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:24.771+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:25.728 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:25.521+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:25.728 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:25.521+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:25.728 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:25.521+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:25.728 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:25.521+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:25.728 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:25.409+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:25.728 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:25.409+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:25.728 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:25.409+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:25.728 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:25.409+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:25.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:25.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:25.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:25.727+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:26.753 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:26.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:26.753 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:26.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:26.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:26.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:26.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:26.381+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:26.754 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:26.526+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:26.754 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:26.526+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:26.754 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:26.526+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:26.754 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:26.527+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:27.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:06:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:06:26.972+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 14183 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:06:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:26.751+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:26.751+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:26.751+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:26.752+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:27.707 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:27.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:27.707 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:27.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:27.708 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:27.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:27.708 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:27.410+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:27.708 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:27.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:27.708 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:27.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:27.708 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:27.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:27.708 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:27.569+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:28.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:27.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:28.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:27.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:28.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:27.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:27.776+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:28.615 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:28.363+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:28.615 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:28.363+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:28.615 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:28.363+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:28.615 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:28.364+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:28.615 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:28.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:28.615 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:28.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:28.615 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:28.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:28.615 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:28.613+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:29.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:28.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:29.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:28.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:28.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:28.821+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:29.660 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:29.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:29.661 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:29.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:29.661 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:29.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:29.661 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:29.405+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:29.788+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:29.788+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:29.788+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:29.789+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:29.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:29.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:29.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:29.660+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:30.685 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:30.421+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:30.685 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:30.421+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:30.685 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:30.421+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:30.685 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:30.421+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:30.758+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:30.758+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:30.758+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:30.759+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:30.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:30.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:30.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:30.684+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:31.645 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:31.379+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:31.646 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:31.379+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:31.646 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:31.379+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:31.646 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:31.380+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:31.975 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:31.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:31.975 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:31.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:31.975 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:31.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:31.975 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:31.714+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:31.976 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:31.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:31.976 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:31.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:31.976 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:31.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:31.976 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:31.645+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:32.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:06:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:06:31.973+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 14242 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:06:32.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:32.356+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:32.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:32.356+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:32.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:32.356+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:32.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:32.357+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:32.672+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:32.672+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:32.672+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:32.673+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:32.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:32.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:32.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:32.664+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:33.680 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:33.406+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:33.680 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:33.406+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:33.680 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:33.406+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:33.680 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:33.407+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:34.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:33.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:33.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:33.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:33.679+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:33.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:33.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:33.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:33.718+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:34.687 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:34.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:34.687 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:34.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:34.687 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:34.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:34.687 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:34.371+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:35.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:34.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:34.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:34.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:34.687+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:34.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:34.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:34.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:34.703+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:35.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:35.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:35.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:35.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:35.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:35.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:35.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:35.371+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:35.694+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:35.694+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:35.694+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:35.695+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:35.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:35.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:35.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:35.685+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:36.706 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:36.419+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:36.706 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:36.419+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:36.706 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:36.419+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:36.706 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:36.420+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:36.976 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:36.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:36.976 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:36.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:36.976 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:36.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:36.976 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:36.705+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:36.977 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:36.735+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:36.977 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:36.735+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:36.977 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:36.735+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:36.977 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:36.736+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:37.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:06:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:06:36.974+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 14300 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:06:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:37.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:37.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:37.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:37.662+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:37.453+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:37.453+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:37.453+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:37.454+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:37.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:37.688+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:37.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:37.688+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:37.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:37.688+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:37.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:37.689+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:38.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:38.681+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:38.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:38.681+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:38.681+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:38.682+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:38.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:38.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:38.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:38.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:38.449+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:38.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:38.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:38.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:38.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:38.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:38.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:38.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:38.650+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:39.661 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:39.406+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:39.661 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:39.406+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:39.661 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:39.406+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:39.661 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:39.407+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:39.662 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:39.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:39.662 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:39.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:39.662 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:39.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:39.662 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:39.639+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:40.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:39.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:39.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:39.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:39.661+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:40.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:40.456+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:40.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:40.456+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:40.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:40.456+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:40.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:40.457+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:40.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:40.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:40.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:40.626+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:40.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:40.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:40.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:40.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:40.657+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:41.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:41.656+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:41.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:41.656+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:41.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:41.656+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:41.656+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:41.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:41.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:41.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:41.437+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:41.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:41.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:41.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:41.682+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:42.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:06:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:06:41.976+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 14359 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:06:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:42.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:42.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:42.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:42.618+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:42.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:42.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:42.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:42.411+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:42.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:42.640+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:42.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:42.640+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:42.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:42.640+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:42.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:42.641+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:43.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:43.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:43.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:43.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:43.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:43.411+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:44.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:43.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:44.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:43.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:43.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:43.628+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:43.606+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:43.606+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:43.606+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:43.606+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:44.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:44.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:44.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:44.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:44.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:44.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:44.395+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:44.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:44.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:44.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:44.617+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:44.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:44.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:44.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:44.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:44.609+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:45.732 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:45.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:45.732 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:45.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:45.733 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:45.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:45.733 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:45.656+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:45.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:45.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:45.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:45.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:45.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:45.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:45.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:45.647+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:45.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:45.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:45.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:45.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:45.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:45.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:45.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:45.413+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:46.630 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:46.388+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:46.631 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:46.388+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:46.631 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:46.388+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:46.631 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:46.389+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:46.979 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:46.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:46.980 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:46.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:46.980 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:46.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:46.980 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:46.685+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:46.980 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:46.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:46.980 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:46.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:46.980 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:46.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:46.980 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:46.629+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:47.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:06:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:06:46.977+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 14418 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:06:47.727 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:47.429+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:47.727 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:47.429+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:47.727 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:47.429+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:47.727 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:47.429+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:47.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:47.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:47.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:47.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:47.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:47.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:47.728 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:47.661+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:48.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:47.725+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:48.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:47.725+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:48.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:47.725+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:48.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:47.725+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:48.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:48.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:48.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:48.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:48.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:48.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:48.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:48.427+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:48.750 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:48.621+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:48.750 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:48.621+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:48.750 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:48.621+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:48.750 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:48.621+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:49.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:48.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:49.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:48.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:48.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:48.748+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:49.768 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:49.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:49.768 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:49.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:49.768 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:49.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:49.768 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:49.669+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:49.768 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:49.458+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:49.769 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:49.458+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:49.769 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:49.458+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:49.769 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:49.459+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:50.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:49.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:50.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:49.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:50.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:49.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:50.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:49.767+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:50.813 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:50.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:50.814 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:50.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:50.814 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:50.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:50.814 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:50.667+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:50.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:50.463+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:50.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:50.463+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:50.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:50.463+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:50.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:50.463+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:51.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:50.812+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:50.812+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:50.812+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:50.813+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:51.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:51.619+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:51.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:51.619+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:51.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:51.619+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:51.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:51.620+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:51.782 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:51.465+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:51.782 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:51.465+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:51.782 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:51.465+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:51.782 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:51.465+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:52.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:06:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:06:51.979+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 14476 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:06:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:51.779+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:51.779+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:51.779+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:51.780+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:52.748 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:52.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:52.748 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:52.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:52.749 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:52.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:52.749 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:52.598+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:52.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:52.456+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:52.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:52.456+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:52.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:52.456+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:52.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:52.456+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:53.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:52.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:52.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:52.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:52.748+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:53.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:53.618+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:53.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:53.618+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:53.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:53.618+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:53.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:53.618+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:53.728 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:53.434+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:53.728 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:53.434+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:53.728 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:53.434+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:53.728 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:53.435+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:54.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:53.725+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:54.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:53.725+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:53.725+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:53.726+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:54.701+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:54.701+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:54.701+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:54.702+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:54.576+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:54.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:54.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:54.577+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:54.461+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:54.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:54.461+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:54.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:54.461+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:54.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:54.461+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:55.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:55.462+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:55.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:55.462+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:55.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:55.462+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:55.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:55.463+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:55.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:55.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:55.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:55.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:55.688+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:55.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:55.573+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:55.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:55.573+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:55.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:55.573+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:55.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:55.574+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:56.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:56.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:56.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:56.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:56.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:56.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:56.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:56.571+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:56.729 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:56.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:56.729 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:56.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:56.730 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:56.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:56.730 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:56.450+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:56.982 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:56.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:56.982 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:56.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:56.982 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:56.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:56.982 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:56.728+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:57.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:06:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:06:56.980+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 14534 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:06:57.775 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:57.489+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:57.775 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:57.489+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:57.775 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:57.489+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:57.775 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:57.490+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:57.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:57.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:57.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:57.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:57.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:57.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:57.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:57.536+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:58.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:57.773+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:57.773+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:57.773+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:57.774+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:58.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:58.441+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:58.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:58.441+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:58.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:58.441+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:58.555 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:58.441+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:58.791+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:58.791+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:58.791+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:06:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:58.792+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:58.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:58.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:58.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:58.554+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:06:59.745 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:59.467+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:59.746 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:59.467+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:59.746 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:59.467+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:06:59.746 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:06:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:06:59.467+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:06:59.746 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:59.583+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:59.746 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:59.583+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:59.746 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:59.583+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:06:59.746 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:06:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:06:59.584+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:00.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:59.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:00.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:59.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:00.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:59.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:00.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:06:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:06:59.744+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:00.721 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:00.468+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:00.721 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:00.468+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:00.721 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:00.468+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:00.721 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:00.468+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:00.722 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:00.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:00.722 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:00.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:00.722 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:00.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:00.722 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:00.610+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:00.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:00.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:00.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:00.720+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:01.746 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:01.574+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:01.746 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:01.574+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:01.746 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:01.574+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:01.746 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:01.575+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:01.747 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:01.435+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:01.747 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:01.435+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:01.747 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:01.435+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:01.747 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:01.436+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:02.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:07:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:07:01.982+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 14593 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:07:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:01.745+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:01.745+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:01.745+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:01.746+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:02.775 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:02.465+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:02.775 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:02.465+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:02.776 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:02.465+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:02.776 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:02.465+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:02.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:02.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:02.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:02.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:02.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:02.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:02.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:02.596+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:02.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:02.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:02.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:02.774+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:03.611 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:03.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:03.611 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:03.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:03.611 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:03.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:03.611 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:03.592+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:03.611 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:03.450+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:03.611 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:03.450+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:03.611 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:03.450+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:03.611 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:03.451+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:04.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:03.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:04.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:03.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:04.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:03.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:04.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:03.809+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:04.800 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:04.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:04.800 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:04.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:04.801 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:04.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:04.801 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:04.545+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:04.801 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:04.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:04.801 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:04.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:04.801 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:04.412+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:04.801 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:04.413+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:05.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:04.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:05.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:04.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:05.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:04.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:04.836+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:05.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:05.567+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:05.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:05.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:05.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:05.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:05.568+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:05.398+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:05.398+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:05.398+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:05.399+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:06.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:05.859+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:06.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:05.859+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:05.859+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:05.860+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:06.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:06.597+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:06.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:06.597+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:06.597+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:06.598+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:06.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:06.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:06.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:06.372+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:07.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:06.860+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:06.860+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:06.860+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:06.861+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:07.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:07:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:07:06.983+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 14652 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:07:07.616 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:07.335+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:07.616 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:07.335+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:07.616 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:07.335+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:07.616 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:07.336+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:07.873 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:07.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:07.873 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:07.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:07.873 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:07.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:07.873 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:07.615+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:08.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:07.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:08.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:07.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:08.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:07.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:07.884+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:08.651 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:08.384+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:08.652 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:08.384+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:08.652 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:08.384+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:08.652 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:08.385+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:09.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:08.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:09.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:08.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:08.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:08.650+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:08.880+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:08.880+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:08.880+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:08.881+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:09.628 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:09.343+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:09.628 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:09.343+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:09.628 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:09.343+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:09.628 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:09.344+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:09.880 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:09.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:09.880 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:09.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:09.880 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:09.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:09.880 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:09.627+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:10.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:09.878+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:10.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:09.878+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:10.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:09.878+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:09.879+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:10.661 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:10.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:10.661 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:10.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:10.661 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:10.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:10.661 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:10.362+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:10.945 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:10.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:10.945 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:10.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:10.945 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:10.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:10.945 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:10.838+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:10.946 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:10.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:10.946 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:10.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:10.946 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:10.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:10.946 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:10.660+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:11.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:11.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:11.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:11.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:11.613+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:11.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:11.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:11.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:11.400+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:12.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:07:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:07:11.985+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 14711 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:07:12.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:11.881+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:11.881+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:11.881+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:11.882+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:12.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:12.431+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:12.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:12.431+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:12.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:12.431+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:12.432+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:12.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:12.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:12.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:12.582+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:13.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:12.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:13.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:12.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:12.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:12.843+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:13.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:13.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:13.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:13.611+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:13.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:13.422+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:13.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:13.422+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:13.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:13.422+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:13.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:13.423+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:14.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:13.880+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:14.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:13.880+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:13.880+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:13.880+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:14.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:14.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:14.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:14.587+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:14.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:14.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:14.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:14.420+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:14.910+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:14.910+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:14.910+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:14.910+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:15.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:15.387+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:15.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:15.387+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:15.387+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:15.388+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:15.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:15.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:15.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:15.567+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:16.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:15.864+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:15.864+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:15.864+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:15.864+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:16.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:16.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:16.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:16.592+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:16.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:16.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:16.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:16.354+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:17.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:16.849+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:17.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:16.849+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:17.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:16.849+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:17.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:16.849+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:17.309 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:07:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:07:16.986+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 14769 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:07:17.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:17.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:17.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:17.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:17.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:17.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:17.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:17.308+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:17.877 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:17.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:17.877 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:17.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:17.877 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:17.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:17.877 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:17.604+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:17.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:17.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:17.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:17.876+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:18.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:18.564+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:18.564+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:18.564+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:18.565+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:18.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:18.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:18.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:18.352+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:19.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:18.858+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:18.858+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:18.858+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:18.859+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:19.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:19.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:19.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:19.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:19.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:19.400+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:19.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:19.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:19.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:19.577+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:20.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:19.886+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:20.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:19.886+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:20.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:19.886+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:19.886+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:20.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:20.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:20.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:20.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:20.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:20.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:20.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:20.375+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:20.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:20.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:20.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:20.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:20.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:20.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:20.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:20.586+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:21.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:20.852+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:21.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:20.852+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:21.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:20.852+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:21.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:20.852+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:21.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:21.403+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:21.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:21.403+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:21.403+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:21.404+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:21.575+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:21.575+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:21.575+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:21.576+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:22.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:07:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:07:21.987+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 14828 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:07:22.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:21.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:21.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:21.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:21.836+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:22.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:22.398+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:22.812 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:22.398+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:22.812 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:22.398+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:22.812 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:22.399+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:22.812 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:22.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:22.812 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:22.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:22.812 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:22.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:22.813 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:22.543+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:22.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:22.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:22.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:22.809+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:23.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:23.442+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:23.442+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:23.442+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:23.442+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:24.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:23.807+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:24.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:23.807+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:24.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:23.807+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:23.808+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:23.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:23.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:23.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:23.591+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:24.807 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:24.438+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:24.807 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:24.438+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:24.807 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:24.438+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:24.807 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:24.439+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:24.808 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:24.583+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:24.808 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:24.583+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:24.808 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:24.583+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:24.808 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:24.583+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:25.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:24.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:25.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:24.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:25.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:24.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:24.806+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:25.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:25.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:25.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:25.620+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:25.417+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:25.417+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:25.417+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:25.418+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:26.304 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:25.841+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:26.304 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:25.841+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:26.305 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:25.841+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:26.305 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:25.842+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:26.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:26.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:26.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:26.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:26.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:26.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:26.379+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:26.872 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:26.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:26.873 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:26.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:26.873 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:26.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:26.873 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:26.587+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:27.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:07:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:07:26.989+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 14887 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:07:27.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:26.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:26.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:26.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:26.872+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:27.618 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:27.356+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:27.618 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:27.356+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:27.618 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:27.356+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:27.618 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:27.357+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:27.882 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:27.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:27.882 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:27.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:27.883 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:27.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:27.883 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:27.617+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:28.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:27.881+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:28.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:27.881+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:27.881+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:27.882+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:28.801 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:28.395+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:28.802 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:28.395+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:28.802 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:28.395+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:28.802 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:28.395+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:28.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:28.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:28.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:28.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:28.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:28.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:28.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:28.590+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:28.876+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:28.876+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:28.876+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:28.876+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:29.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:29.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:29.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:29.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:29.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:29.390+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:29.579+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:29.579+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:29.579+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:29.580+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:30.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:29.889+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:30.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:29.889+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:30.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:29.889+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:30.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:29.890+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:30.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:30.393+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:30.393+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:30.393+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:30.393+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:30.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:30.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:30.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:30.553+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:31.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:30.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:31.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:30.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:30.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:30.884+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:31.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:31.365+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:31.365+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:31.365+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:31.366+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:31.518+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:31.518+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:31.518+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:31.519+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:32.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:31.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:31.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:31.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:31.903+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:32.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:07:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:07:31.991+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 14946 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:07:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:32.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:32.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:32.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:32.552+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:32.352+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:32.352+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:32.352+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:32.353+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:32.863+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:32.863+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:32.863+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:32.863+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:33.612 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:33.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:33.613 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:33.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:33.613 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:33.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:33.613 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:33.591+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:33.613 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:33.355+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:33.613 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:33.355+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:33.613 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:33.355+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:33.613 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:33.355+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:34.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:33.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:34.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:33.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:34.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:33.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:34.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:33.872+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:34.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:34.319+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:34.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:34.319+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:34.319+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:34.320+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:34.873 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:34.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:34.873 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:34.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:34.873 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:34.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:34.873 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:34.620+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:35.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:34.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:35.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:34.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:35.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:34.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:34.872+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:35.644 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:35.336+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:35.645 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:35.336+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:35.645 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:35.336+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:35.645 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:35.336+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:35.916 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:35.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:35.916 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:35.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:35.916 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:35.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:35.916 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:35.644+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:36.314 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:35.914+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:36.315 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:35.914+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:36.315 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:35.914+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:36.315 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:35.915+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:36.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:36.313+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:36.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:36.313+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:36.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:36.313+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:36.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:36.314+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:36.896 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:36.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:36.896 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:36.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:36.896 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:36.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:36.896 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:36.617+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:37.307 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:07:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:07:36.992+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 15004 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:07:37.307 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:36.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:37.307 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:36.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:37.307 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:36.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:37.307 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:36.895+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:37.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:37.305+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:37.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:37.305+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:37.305+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:37.306+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:37.914 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:37.579+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:37.915 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:37.579+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:37.915 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:37.579+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:37.915 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:37.580+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:38.295 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:37.913+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:38.295 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:37.913+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:38.295 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:37.913+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:38.295 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:37.914+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:38.422 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T04:07:38.419+0000 7f12908b2640 0 monclient(hunting): authenticate timed out after 300 2024-09-10T04:07:38.422 INFO:teuthology.orchestra.run.smithi112.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-09-10T04:07:38.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:38.293+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:38.293+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:38.293+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:38.294+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:38.972 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:38.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:38.972 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:38.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:38.972 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:38.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:38.972 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:38.872+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:38.972 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:38.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:38.973 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:38.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:38.973 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:38.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:38.973 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:38.628+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:39.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T04:07:39.192 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephadm.py", line 1525, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephadm.py", line 111, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi112 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -e sha1=3a5e3f16720f432aea965c40a8f6eb480095ec6c -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-09-10T04:07:39.443 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3ebaa8e069d84b159680a2a04088c8c7 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephadm.py", line 1525, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephadm.py", line 111, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi112 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -e sha1=3a5e3f16720f432aea965c40a8f6eb480095ec6c -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-09-10T04:07:39.446 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-10T04:07:39.471 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-10T04:07:39.471 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T04:07:39.472 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T04:07:39.493 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T04:07:39.493 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T04:07:39.549 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd blocklist ls 2024-09-10T04:07:39.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:39.300+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:39.300+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:39.300+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:39.300+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:39.841 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T04:07:39.965 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:39.831+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:39.965 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:39.831+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:39.965 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:39.831+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:39.965 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:39.832+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:39.965 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:39.642+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:39.966 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:39.642+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:39.966 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:39.642+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:39.966 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:39.642+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:40.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:40.323+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:40.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:40.323+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:40.323+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:40.324+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:41.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:40.788+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:40.788+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:40.788+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:40.789+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:40.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:40.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:40.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:40.666+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:41.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:41.296+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:41.296+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:41.296+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:41.297+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:41.996 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:41.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:41.996 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:41.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:41.996 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:41.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:41.996 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:41.769+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:41.997 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:41.716+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:41.997 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:41.716+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:41.997 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:41.716+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:41.997 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:41.717+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:42.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:07:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:07:41.994+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 15063 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:07:42.669 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:42.340+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:42.669 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:42.340+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:42.669 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:42.340+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:42.669 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:42.340+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:43.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:42.667+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:43.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:42.667+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:42.667+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:42.668+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:42.804+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:42.804+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:42.804+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:42.805+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:43.614 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:43.332+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:43.614 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:43.332+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:43.614 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:43.332+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:43.614 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:43.333+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:44.041 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:43.812+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:44.041 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:43.812+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:44.042 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:43.812+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:44.042 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:43.813+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:44.042 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:43.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:44.042 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:43.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:44.042 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:43.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:44.042 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:43.623+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:44.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:44.357+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:44.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:44.357+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:44.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:44.357+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:44.599 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:44.358+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:45.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:44.597+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:45.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:44.597+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:45.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:44.597+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:45.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:44.598+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:44.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:44.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:45.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:44.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:45.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:44.817+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:45.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:45.326+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:45.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:45.326+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:45.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:45.326+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:45.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:45.327+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:46.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:45.826+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:45.826+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:45.826+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:45.827+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:45.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:45.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:45.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:46.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:45.613+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:46.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:46.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:46.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:46.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:46.280+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:46.997 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:46.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:46.997 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:46.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:46.997 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:46.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:46.998 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:46.614+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:46.998 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:46.808+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:46.998 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:46.808+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:46.998 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:46.808+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:46.999 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:46.809+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:47.286 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:07:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:07:46.996+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 15122 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:07:47.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:47.285+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:47.285+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:47.285+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:47.286+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:48.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:47.831+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:47.831+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:47.831+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:47.832+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:47.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:47.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:47.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:48.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:47.591+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:48.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:48.250+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:48.250+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:48.250+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:48.251+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:48.847+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:48.847+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:48.847+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:48.847+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:48.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:49.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:48.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:49.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:48.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:49.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:48.601+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:49.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:49.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:49.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:49.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:49.281+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:50.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:49.848+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:49.848+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:49.848+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:49.848+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:49.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:49.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:49.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:50.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:49.623+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:50.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:50.316+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:50.316+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:50.316+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:50.317+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:51.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:50.834+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:50.834+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:50.834+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:50.835+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:51.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:50.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:51.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:50.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:51.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:50.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:51.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:50.647+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:51.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:51.270+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:51.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:51.270+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:51.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:51.270+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:51.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:51.271+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:51.999 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:51.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:51.999 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:51.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:51.999 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:51.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:51.999 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:51.648+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:52.000 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:51.870+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:52.000 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:51.870+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:52.000 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:51.870+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:52.000 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:51.871+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:52.280 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:07:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:07:51.997+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 15181 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:07:52.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:52.279+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:52.279+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:52.279+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:52.279+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:53.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:52.885+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:53.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:52.885+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:52.885+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:52.885+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:53.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:52.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:53.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:52.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:53.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:52.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:53.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:52.650+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:53.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:53.322+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:53.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:53.322+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:53.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:53.322+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:53.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:53.322+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:53.932 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:53.635+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:53.933 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:53.635+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:53.933 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:53.635+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:53.933 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:53.635+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:54.277 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:53.931+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:54.277 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:53.931+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:54.277 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:53.931+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:54.277 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:53.932+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:54.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:54.324+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:54.324+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:54.324+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:54.324+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:54.884 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:54.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:54.884 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:54.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:54.885 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:54.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:54.885 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:54.592+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:55.154 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:54.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:55.154 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:54.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:55.155 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:54.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:55.155 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:54.883+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:55.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:55.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:55.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:55.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:55.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:55.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:55.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:55.300+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:55.906 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:55.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:55.906 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:55.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:55.906 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:55.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:55.906 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:55.638+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:56.257 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:55.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:56.257 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:55.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:56.257 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:55.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:56.257 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:55.905+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:56.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:56.255+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:56.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:56.255+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:56.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:56.255+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:56.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:56.256+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:56.912 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:56.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:56.913 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:56.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:56.913 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:56.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:56.913 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:56.617+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:57.251 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:07:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:07:56.998+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 15239 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:07:57.251 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:56.910+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:57.252 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:56.910+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:57.252 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:56.910+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:57.252 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:56.911+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:57.572 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:57.249+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:57.572 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:57.249+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:57.572 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:57.249+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:57.572 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:57.249+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:57.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:57.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:57.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:57.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:57.570+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:58.224 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:57.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:58.224 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:57.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:58.224 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:57.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:58.224 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:57.954+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:58.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:58.221+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:58.221+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:58.221+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:58.222+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:58.996 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:58.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:58.997 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:58.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:58.997 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:58.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:58.997 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:58.589+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:07:59.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:58.994+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:58.994+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:58.994+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:07:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:58.995+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:59.217+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:59.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:59.217+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:59.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:59.217+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:07:59.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:07:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:07:59.217+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:07:59.977 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:59.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:59.977 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:59.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:59.977 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:59.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:07:59.978 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:07:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:07:59.604+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:00.244 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:59.974+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:00.244 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:59.974+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:00.244 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:59.974+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:00.244 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:07:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:07:59.975+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:00.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:00.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:00.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:00.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:00.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:00.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:00.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:00.242+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:00.990 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:00.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:00.990 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:00.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:00.990 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:00.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:00.990 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:00.589+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:01.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:00.987+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:01.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:00.987+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:00.987+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:00.988+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:01.195+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:01.195+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:01.195+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:01.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:01.196+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:01.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:01.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:01.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:01.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:01.542+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:02.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:08:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:08:01.999+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 15298 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:08:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:02.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:02.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:02.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:02.149+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:02.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:02.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:02.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:02.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:02.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:02.008+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:02.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:02.501+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:02.501+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:02.501+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:02.501+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:03.176+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:03.176+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:03.176+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:03.177+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:03.003+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:03.003+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:03.003+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:03.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:03.004+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:03.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:03.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:03.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:03.457+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:04.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:03.959+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:03.959+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:03.959+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:03.960+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:04.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:04.188+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:04.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:04.188+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:04.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:04.188+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:04.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:04.189+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:04.451+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:04.451+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:04.451+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:04.452+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:04.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:04.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:04.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:04.993+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:05.201+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:05.202+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:05.202+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:05.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:05.202+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:05.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:05.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:05.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:05.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:05.437+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:06.159+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:06.159+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:06.159+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:06.160+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:06.042+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:06.042+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:06.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:06.042+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:06.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:06.043+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:06.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:06.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:06.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:06.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:06.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:06.455+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:07.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:07.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:07.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:07.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:07.019+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:07.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:07.206+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:07.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:07.206+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:07.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:07.206+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:07.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:07.207+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:07.332 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:08:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:08:07.002+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 15357 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:08:07.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:07.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:07.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:07.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:07.414+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:08.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:08.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:08.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:08.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:08.005+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:08.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:08.220+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:08.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:08.220+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:08.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:08.220+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:08.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:08.221+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:08.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:08.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:08.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:08.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:08.414+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:09.313 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:08.982+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:09.313 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:08.982+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:09.314 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:08.982+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:09.314 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:08.983+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:09.314 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:09.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:09.314 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:09.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:09.314 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:09.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:09.314 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:09.198+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:09.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:09.396+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:09.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:09.396+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:09.396+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:09.396+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:10.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:10.031+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:10.032+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:10.032+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:10.032+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:10.229+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:10.229+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:10.229+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:10.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:10.230+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:10.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:10.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:10.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:10.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:10.418+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:11.246+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:11.246+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:11.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:11.246+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:11.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:11.247+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:11.051+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:11.051+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:11.051+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:11.052+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:11.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:11.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:11.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:11.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:11.457+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:12.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:08:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:08:12.004+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 15416 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:08:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:12.028+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:12.028+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:12.028+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:12.029+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:12.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:12.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:12.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:12.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:12.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:12.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:12.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:12.243+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:12.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:12.440+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:12.440+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:12.440+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:12.441+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:13.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:13.237+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:13.237+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:13.237+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:13.238+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:13.070+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:13.070+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:13.070+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:13.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:13.071+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:13.465+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:13.465+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:13.465+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:13.466+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:14.482 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:14.110+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:14.482 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:14.110+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:14.482 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:14.110+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:14.482 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:14.111+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:14.483 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:14.278+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:14.483 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:14.278+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:14.483 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:14.278+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:14.483 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:14.278+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:14.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:14.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:14.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:14.481+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:15.457 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:15.116+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:15.596 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:15.116+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:15.597 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:15.116+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:15.597 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:15.116+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:15.597 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:15.253+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:15.597 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:15.253+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:15.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:15.253+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:15.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:15.254+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:15.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:16.423 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:15.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:16.424 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:15.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:16.424 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:15.525+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:16.425 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:16.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:16.425 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:16.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:16.425 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:16.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:16.425 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:16.281+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:16.425 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:16.069+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:16.425 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:16.069+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:16.426 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:16.069+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:16.426 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:16.069+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:16.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:16.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:16.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:16.553+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:17.300 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:17.048+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:17.300 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:17.048+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:17.300 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:17.048+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:17.300 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:17.048+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:17.300 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:08:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:08:17.005+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 15474 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:08:17.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:17.298+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:17.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:17.298+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:17.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:17.298+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:17.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:17.299+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:17.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:17.567+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:17.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:17.567+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:17.567+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:17.567+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:18.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:18.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:18.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:18.049+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:18.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:18.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:18.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:18.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:18.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:18.549+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:18.331+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:18.331+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:18.331+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:18.331+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:19.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:19.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:19.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:19.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:19.063+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:19.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:19.304+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:19.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:19.304+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:19.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:19.304+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:19.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:19.304+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:19.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:19.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:19.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:19.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:19.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:19.504+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:20.281 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:20.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:20.282 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:20.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:20.282 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:20.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:20.282 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:20.016+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:20.534 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:20.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:20.534 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:20.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:20.534 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:20.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:20.534 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:20.281+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:20.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:20.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:20.532+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:20.533+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:21.272 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:20.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:21.272 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:20.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:21.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:20.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:21.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:20.976+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:21.558 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:21.270+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:21.559 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:21.270+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:21.559 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:21.270+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:21.560 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:21.271+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:21.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:21.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:21.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:21.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:21.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:21.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:21.557+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:22.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:08:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:08:22.007+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 15533 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:08:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:22.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:22.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:22.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:22.017+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:22.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:22.240+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:22.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:22.240+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:22.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:22.240+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:22.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:22.241+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:23.028 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:22.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:23.028 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:22.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:23.028 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:22.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:23.029 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:22.603+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:23.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:23.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:23.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:23.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:23.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:23.027+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:23.268+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:23.268+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:23.268+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:23.269+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:23.992 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:23.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:23.992 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:23.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:23.992 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:23.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:23.992 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:23.601+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:24.312 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:23.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:24.313 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:23.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:24.313 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:23.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:24.313 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:23.991+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:24.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:24.311+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:24.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:24.311+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:24.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:24.311+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:24.569 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:24.312+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:24.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:24.567+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:24.567+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:24.567+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:24.568+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:25.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:24.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:25.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:24.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:25.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:24.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:25.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:24.947+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:25.553 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:25.269+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:25.554 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:25.269+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:25.554 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:25.269+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:25.554 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:25.270+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:25.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:25.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:25.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:25.553+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:26.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:25.951+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:26.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:25.951+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:26.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:25.951+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:26.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:25.952+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:26.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:26.269+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:26.269+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:26.269+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:26.270+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:26.941 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:26.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:26.941 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:26.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:26.941 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:26.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:26.942 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:26.597+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:27.237 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:08:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:08:27.008+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 15592 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:08:27.237 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:26.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:27.237 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:26.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:27.237 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:26.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:27.238 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:26.940+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:27.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:27.235+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:27.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:27.235+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:27.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:27.235+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:27.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:27.236+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:27.932 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:27.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:27.932 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:27.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:27.932 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:27.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:27.933 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:27.585+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:28.224 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:27.931+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:28.225 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:27.931+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:28.225 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:27.931+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:28.225 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:27.931+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:28.564 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:28.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:28.565 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:28.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:28.565 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:28.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:28.565 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:28.223+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:28.563+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:28.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:28.563+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:28.563+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:28.564+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:29.224 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:28.949+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:29.225 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:28.949+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:29.225 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:28.949+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:29.225 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:28.949+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:29.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:29.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:29.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:29.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:29.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:29.224+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:29.861 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:29.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:29.862 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:29.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:29.862 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:29.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:29.862 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:29.587+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:30.220 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:29.924+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:30.220 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:29.924+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:30.220 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:29.924+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:30.220 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:29.925+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:30.562 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:30.218+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:30.562 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:30.218+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:30.562 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:30.218+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:30.562 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:30.219+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:30.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:30.560+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:30.560+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:30.560+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:30.561+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:31.173 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:30.889+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:31.173 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:30.889+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:31.173 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:30.889+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:31.174 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:30.890+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:31.530 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:31.172+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:31.530 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:31.172+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:31.530 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:31.172+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:31.530 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:31.172+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:31.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:31.528+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:31.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:31.528+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:31.528+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:31.529+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:32.215 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:08:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:08:32.010+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 15660 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:08:32.215 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:31.923+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:32.215 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:31.923+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:32.215 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:31.923+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:32.215 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:31.924+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:32.547 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:32.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:32.547 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:32.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:32.547 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:32.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:32.548 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:32.214+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:32.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:32.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:32.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:32.546+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:33.182 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:32.895+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:33.183 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:32.895+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:33.183 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:32.895+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:33.183 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:32.896+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:33.572 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:33.181+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:33.572 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:33.181+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:33.572 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:33.181+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:33.573 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:33.182+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:33.570+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:33.570+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:33.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:33.571+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:34.145 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:33.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:34.146 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:33.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:34.146 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:33.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:34.146 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:33.904+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:34.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:34.143+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:34.143+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:34.143+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:34.144+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:34.918 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:34.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:34.918 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:34.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:34.918 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:34.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:34.918 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:34.587+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:35.168 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:34.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:35.169 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:34.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:35.169 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:34.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:35.169 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:34.916+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:35.169 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:35.165+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:35.169 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:35.165+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:35.169 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:35.165+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:35.170 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:35.166+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:35.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:35.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:35.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:35.555+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:36.194 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:35.889+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:36.195 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:35.889+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:36.195 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:35.889+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:36.195 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:35.890+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:36.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:36.192+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:36.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:36.192+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:36.192+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:36.193+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:36.871 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:36.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:36.871 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:36.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:36.871 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:36.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:36.872 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:36.589+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:37.240 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:08:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:08:37.011+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 15718 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:08:37.240 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:36.869+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:37.241 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:36.869+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:37.241 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:36.869+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:37.241 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:36.869+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:37.553 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:37.237+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:37.554 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:37.237+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:37.554 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:37.237+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:37.554 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:37.238+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:37.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:37.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:37.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:37.552+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:38.206 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:37.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:38.206 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:37.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:38.206 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:37.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:38.207 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:37.866+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:38.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:38.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:38.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:38.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:38.204+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:38.890 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:38.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:38.890 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:38.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:38.890 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:38.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:38.890 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:38.589+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:39.253 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:38.887+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:39.253 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:38.887+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:39.253 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:38.887+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:39.253 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:38.888+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:39.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:39.249+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:39.249+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:39.249+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:39.249+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:40.011 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:39.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:40.011 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:39.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:40.011 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:39.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:40.011 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:39.620+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:40.011 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:39.863+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:40.012 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:39.863+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:40.012 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:39.863+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:40.012 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:39.864+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:40.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:40.278+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:40.278+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:40.278+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:40.279+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:41.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:40.847+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:40.847+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:40.847+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:40.848+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:40.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:40.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:40.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:41.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:40.608+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:41.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:41.268+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:41.269+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:41.269+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:41.269+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:42.015 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:41.836+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:42.016 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:41.836+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:42.016 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:41.836+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:42.016 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:41.837+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:42.016 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:41.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:42.016 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:41.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:42.016 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:41.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:42.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:41.603+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:42.315 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:08:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:08:42.014+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 15777 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:08:42.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:42.314+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:42.314+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:42.314+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:42.315+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:43.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:42.822+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:42.822+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:42.822+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:42.823+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:42.597+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:42.597+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:43.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:42.597+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:43.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:42.598+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:43.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:43.306+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:43.306+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:43.306+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:43.307+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:43.844 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:43.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:43.844 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:43.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:43.845 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:43.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:43.845 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:43.582+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:44.296 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:43.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:44.296 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:43.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:44.296 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:43.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:44.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:43.843+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:44.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:44.539+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:44.539+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:44.539+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:44.540+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:44.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:44.294+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:44.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:44.294+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:44.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:44.294+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:44.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:44.295+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:45.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:44.813+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:44.813+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:44.813+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:44.814+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:45.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:45.293+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:45.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:45.293+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:45.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:45.293+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:45.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:45.294+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:46.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:45.788+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:45.788+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:45.788+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:45.788+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:45.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:45.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:45.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:46.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:45.582+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:46.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:46.269+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:46.269+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:46.269+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:46.270+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:47.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:46.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:47.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:46.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:47.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:46.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:47.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:46.610+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:47.018 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:46.792+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:47.018 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:46.792+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:47.018 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:46.792+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:47.018 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:46.793+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:47.288 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:08:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:08:47.015+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 15836 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:08:47.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:47.286+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:47.286+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:47.286+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:47.287+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:48.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:47.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:47.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:47.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:47.810+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:47.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:47.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:47.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:48.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:47.642+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:48.261+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:48.261+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:48.261+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:48.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:48.262+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:49.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:48.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:48.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:48.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:48.648+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:48.832+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:48.832+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:48.832+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:49.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:48.832+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:49.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:49.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:49.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:49.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:49.274+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:50.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:49.861+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:49.861+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:49.861+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:49.862+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:49.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:49.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:49.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:49.648+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:50.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:50.236+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:50.236+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:50.236+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:50.236+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:50.867 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:50.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:50.867 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:50.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:50.867 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:50.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:50.867 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:50.606+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:51.275 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:50.865+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:51.275 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:50.865+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:51.275 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:50.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:51.275 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:50.866+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:51.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:51.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:51.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:51.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:51.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:51.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:51.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:51.274+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:52.019 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:51.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:52.019 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:51.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:52.019 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:51.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:52.020 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:51.654+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:52.020 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:51.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:52.020 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:51.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:52.020 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:51.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:52.020 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:51.894+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:52.319 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:08:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:08:52.017+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 15895 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:08:52.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:52.317+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:52.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:52.317+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:52.317+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:52.318+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:52.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:52.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:52.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:52.875+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:53.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:52.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:53.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:52.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:53.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:52.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:53.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:52.641+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:53.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:53.328+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:53.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:53.328+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:53.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:53.328+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:53.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:53.329+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:53.926 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:53.689+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:53.926 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:53.689+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:53.926 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:53.689+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:53.926 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:53.689+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:54.300 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:53.924+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:54.301 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:53.924+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:54.301 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:53.924+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:54.301 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:53.925+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:54.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:54.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:54.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:54.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:54.300+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:54.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:54.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:54.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:54.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:54.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:54.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:54.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:54.704+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:55.276 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:54.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:55.276 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:54.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:55.276 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:54.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:55.277 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:54.954+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:55.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:55.338+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:55.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:55.338+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:55.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:55.338+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:55.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:55.339+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:55.989 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:55.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:55.989 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:55.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:55.989 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:55.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:55.990 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:55.657+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:55.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:55.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:55.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:55.988+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:56.632 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:56.340+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:56.632 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:56.340+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:56.632 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:56.340+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:56.632 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:56.340+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:57.020 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:56.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:57.020 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:56.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:57.021 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:56.630+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:57.021 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:56.631+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:57.329 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:08:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:08:57.018+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 15953 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:08:57.329 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:57.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:57.329 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:57.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:57.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:57.038+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:57.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:57.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:57.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:57.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:57.328+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:58.003 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:57.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:58.004 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:57.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:58.004 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:57.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:58.004 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:57.588+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:58.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:58.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:58.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:58.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:58.002+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:58.632 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:58.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:58.633 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:58.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:58.633 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:58.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:58.633 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:58.371+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:59.039 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:58.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:59.039 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:58.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:59.040 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:58.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:59.040 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:58.632+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:08:59.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:59.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:59.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:59.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:08:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:59.038+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:59.672 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:59.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:59.672 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:59.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:59.672 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:59.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:08:59.673 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:08:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:08:59.401+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:08:59.995 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:59.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:59.996 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:59.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:59.996 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:59.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:08:59.996 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:08:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:08:59.671+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:00.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:59.994+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:00.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:59.994+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:00.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:59.994+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:00.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:08:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:08:59.994+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:00.640 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:00.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:00.641 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:00.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:00.641 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:00.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:00.641 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:00.361+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:01.019 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:00.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:01.019 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:00.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:01.020 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:00.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:01.020 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:00.639+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:01.324 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:01.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:01.324 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:01.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:01.324 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:01.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:01.324 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:01.018+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:01.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:01.322+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:01.322+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:01.322+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:01.323+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:02.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:01.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:02.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:01.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:02.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:01.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:02.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:01.591+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:02.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:02.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:02.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:02.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:02.006+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:02.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:09:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:09:02.020+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16012 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:09:02.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:02.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:02.572+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:02.572+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:02.572+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:02.367+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:02.367+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:02.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:02.367+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:02.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:02.368+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:03.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:03.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:03.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:03.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:03.034+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:03.619 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:03.366+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:03.620 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:03.366+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:03.620 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:03.366+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:03.620 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:03.367+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:03.620 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:03.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:03.620 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:03.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:03.620 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:03.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:03.621 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:03.585+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:04.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:04.026+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:04.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:04.026+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:04.026+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:04.027+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:04.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:04.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:04.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:04.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:04.414+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:04.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:04.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:04.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:04.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:04.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:04.629+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:04.989+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:04.989+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:04.989+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:04.990+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:05.668 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:05.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:05.669 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:05.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:05.669 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:05.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:05.669 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:05.403+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:06.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:05.667+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:06.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:05.667+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:06.018 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:05.667+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:06.018 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:05.667+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:06.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:06.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:06.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:06.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:06.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:06.016+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:06.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:06.429+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:06.429+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:06.429+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:06.430+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:06.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:06.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:06.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:06.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:06.643+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:07.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:09:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:09:07.022+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16071 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:09:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:06.967+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:06.967+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:06.967+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:07.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:06.967+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:07.681 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:07.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:07.681 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:07.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:07.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:07.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:07.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:07.400+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:07.682 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:07.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:07.682 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:07.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:07.682 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:07.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:07.682 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:07.605+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:08.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:07.987+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:08.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:07.987+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:07.987+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:07.987+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:08.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:08.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:08.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:08.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:08.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:08.414+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:08.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:08.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:08.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:08.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:08.617+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:09.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:09.033+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:09.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:09.033+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:09.033+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:09.034+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:09.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:09.430+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:09.430+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:09.430+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:09.431+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:09.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:09.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:09.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:09.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:09.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:09.651+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:10.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:10.060+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:10.060+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:10.060+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:10.060+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:10.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:10.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:10.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:10.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:10.754 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:10.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:10.754 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:10.604+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:10.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:10.393+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:10.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:10.393+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:10.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:10.393+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:10.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:10.394+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:11.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:11.025+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:11.025+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:11.025+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:11.025+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:11.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:11.405+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:11.405+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:11.405+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:11.406+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:11.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:11.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:11.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:11.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:11.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:11.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:11.614+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:12.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:09:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:09:12.022+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16130 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:09:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:12.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:12.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:12.005+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:12.006+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:12.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:12.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:12.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:12.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:12.657+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:12.415+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:12.415+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:12.415+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:12.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:12.416+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:13.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:13.032+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:13.032+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:13.032+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:13.032+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:13.825 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:13.379+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:13.825 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:13.379+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:13.825 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:13.379+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:13.826 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:13.380+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:13.826 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:13.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:13.826 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:13.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:13.827 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:13.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:13.827 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:13.618+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:14.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:14.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:14.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:14.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:14.044+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:14.662 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:14.391+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:14.662 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:14.391+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:14.662 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:14.391+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:14.662 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:14.392+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:15.030 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:14.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:15.030 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:14.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:15.030 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:14.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:15.030 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:14.660+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:15.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:15.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:15.027+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:15.028+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:15.658 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:15.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:15.658 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:15.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:15.659 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:15.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:15.659 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:15.384+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:16.035 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:15.655+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:16.036 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:15.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:16.036 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:15.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:16.036 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:15.656+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:16.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:16.033+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:16.033+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:16.033+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:16.034+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:16.701 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:16.388+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:16.702 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:16.388+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:16.702 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:16.388+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:16.702 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:16.389+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:16.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:16.700+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:16.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:16.700+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:16.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:16.700+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:16.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:16.700+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:17.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:09:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:09:17.025+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16188 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:09:17.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:16.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:16.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:16.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:16.990+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:17.660 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:17.348+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:17.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:17.348+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:17.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:17.348+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:17.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:17.349+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:17.985 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:17.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:17.986 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:17.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:17.986 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:17.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:17.986 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:17.659+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:17.984+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:17.984+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:17.984+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:17.985+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:18.630 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:18.360+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:18.631 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:18.360+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:18.631 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:18.360+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:18.631 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:18.360+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:18.938 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:18.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:18.938 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:18.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:18.938 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:18.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:18.939 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:18.630+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:19.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:18.937+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:18.937+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:18.937+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:18.937+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:19.655 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:19.352+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:19.655 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:19.352+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:19.655 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:19.352+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:19.655 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:19.353+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:19.942 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:19.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:19.942 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:19.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:19.942 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:19.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:19.943 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:19.654+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:20.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:19.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:19.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:19.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:19.941+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:20.639 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:20.373+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:20.639 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:20.373+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:20.639 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:20.373+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:20.640 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:20.374+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:20.949 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:20.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:20.949 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:20.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:20.949 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:20.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:20.949 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:20.637+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:21.329 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:20.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:21.329 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:20.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:21.329 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:20.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:21.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:20.948+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:21.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:21.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:21.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:21.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:21.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:21.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:21.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:21.328+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:21.946 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:21.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:21.947 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:21.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:21.947 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:21.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:21.947 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:21.678+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:22.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:09:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:09:22.026+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16247 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:09:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:21.945+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:21.945+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:21.945+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:21.946+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:22.634 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:22.350+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:22.634 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:22.350+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:22.635 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:22.350+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:22.635 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:22.351+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:22.992 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:22.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:22.992 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:22.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:22.992 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:22.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:22.992 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:22.634+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:23.324 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:22.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:23.325 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:22.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:23.325 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:22.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:23.325 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:22.991+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:23.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:23.323+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:23.323+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:23.323+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:23.323+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:23.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:23.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:23.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:23.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:23.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:23.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:23.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:23.633+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:24.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:23.954+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:23.954+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:23.954+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:23.954+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:24.591 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:24.330+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:24.591 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:24.330+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:24.591 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:24.330+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:24.591 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:24.330+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:24.909 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:24.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:24.909 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:24.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:24.909 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:24.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:24.910 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:24.590+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:25.308 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:24.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:25.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:24.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:25.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:24.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:25.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:24.908+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:25.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:25.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:25.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:25.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:25.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:25.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:25.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:25.307+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:25.906 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:25.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:25.907 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:25.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:25.907 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:25.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:25.907 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:25.625+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:26.295 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:25.905+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:26.295 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:25.905+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:26.295 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:25.905+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:26.296 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:25.906+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:26.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:26.293+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:26.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:26.293+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:26.293+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:26.294+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:27.030 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:26.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:27.030 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:26.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:27.030 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:26.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:27.030 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:26.916+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:27.031 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:26.672+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:27.031 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:26.672+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:27.031 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:26.672+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:27.031 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:26.672+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:27.292 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:09:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:09:27.028+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16306 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:09:27.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:27.291+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:27.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:27.291+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:27.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:27.291+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:27.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:27.292+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:28.086 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:27.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:28.086 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:27.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:28.086 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:27.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:28.086 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:27.905+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:28.087 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:27.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:28.087 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:27.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:28.087 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:27.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:28.087 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:27.662+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:28.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:28.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:28.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:28.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:28.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:28.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:28.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:28.300+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:28.923 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:28.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:28.923 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:28.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:28.923 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:28.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:28.923 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:28.633+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:29.263 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:28.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:29.264 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:28.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:29.264 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:28.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:29.264 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:28.922+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:29.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:29.262+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:29.262+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:29.262+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:29.263+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:29.941 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:29.662+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:29.942 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:29.662+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:29.942 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:29.662+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:29.942 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:29.663+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:30.224 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:29.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:30.224 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:29.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:30.224 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:29.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:30.224 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:29.941+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:30.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:30.222+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:30.222+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:30.222+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:30.223+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:30.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:30.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:30.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:30.651+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:30.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:31.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:30.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:31.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:30.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:31.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:30.895+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:31.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:31.271+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:31.271+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:31.271+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:31.272+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:32.031 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:31.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:32.032 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:31.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:32.032 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:31.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:32.032 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:31.613+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:32.032 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:31.856+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:32.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:31.856+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:32.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:31.856+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:32.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:31.856+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:32.284 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:09:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:09:32.030+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16365 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:09:32.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:32.282+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:32.282+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:32.282+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:32.283+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:33.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:32.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:32.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:32.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:32.651+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:32.836+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:32.836+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:32.836+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:33.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:32.837+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:33.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:33.237+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:33.237+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:33.237+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:33.238+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:34.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:33.792+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:33.792+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:33.792+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:33.793+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:34.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:33.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:34.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:33.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:34.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:33.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:34.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:33.654+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:34.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:34.218+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:34.218+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:34.218+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:34.219+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:35.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:34.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:34.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:34.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:34.671+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:34.786+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:34.786+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:34.786+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:35.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:34.787+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:35.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:35.256+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:35.256+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:35.256+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:35.257+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:35.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:35.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:35.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:35.745+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:35.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:35.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:36.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:35.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:36.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:35.715+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:36.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:36.292+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:36.292+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:36.292+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:36.293+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:37.033 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:36.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:37.034 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:36.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:37.034 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:36.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:37.034 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:36.683+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:37.034 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:36.716+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:37.034 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:36.716+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:37.035 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:36.716+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:37.035 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:36.717+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:37.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:09:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:09:37.031+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16423 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:09:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:37.254+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:37.254+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:37.254+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:37.255+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:37.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:37.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:37.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:37.686+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:38.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:37.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:38.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:37.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:38.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:37.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:38.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:37.748+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:38.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:38.257+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:38.257+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:38.257+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:38.258+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:39.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:38.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:38.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:38.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:38.658+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:38.723+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:38.723+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:39.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:38.723+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:39.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:38.723+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:39.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:39.250+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:39.250+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:39.250+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:39.250+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:40.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:39.672+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:39.672+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:39.672+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:39.673+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:39.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:39.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:40.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:39.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:40.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:39.773+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:40.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:40.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:40.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:40.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:40.214+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:41.046 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:40.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:41.046 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:40.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:41.046 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:40.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:41.046 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:40.820+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:41.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:40.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:41.047 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:40.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:41.047 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:40.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:41.047 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:40.644+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:41.236+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:41.236+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:41.236+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:41.237+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:42.035 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:41.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:42.035 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:41.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:42.035 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:41.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:42.035 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:41.661+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:42.036 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:41.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:42.036 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:41.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:42.036 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:41.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:42.036 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:41.778+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:42.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:09:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:09:42.033+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16482 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:09:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:42.216+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:42.216+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:42.216+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:42.216+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:43.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:42.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:42.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:42.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:42.816+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:42.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:42.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:42.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:43.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:42.705+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:43.191+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:43.191+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:43.191+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:43.192+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:44.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:43.850+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:43.850+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:43.850+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:43.850+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:43.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:43.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:43.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:44.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:43.660+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:44.544 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:44.207+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:44.545 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:44.207+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:44.545 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:44.207+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:44.545 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:44.208+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:44.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:44.655+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:44.655+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:44.655+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:44.656+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:45.174 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:44.852+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:45.174 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:44.852+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:45.174 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:44.852+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:45.174 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:44.853+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:45.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:45.172+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:45.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:45.172+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:45.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:45.172+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:45.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:45.173+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:46.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:45.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:45.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:45.685+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:45.685+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:45.892+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:45.892+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:46.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:45.892+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:46.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:45.892+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:46.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:46.131+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:46.131+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:46.131+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:46.131+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:47.037 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:46.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:47.037 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:46.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:47.037 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:46.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:47.038 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:46.665+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:47.038 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:46.868+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:47.039 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:46.868+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:47.039 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:46.868+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:47.039 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:46.868+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:47.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:09:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:09:47.035+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16541 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:09:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:47.151+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:47.151+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:47.151+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:47.152+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:48.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:47.898+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:47.898+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:47.898+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:47.899+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:47.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:47.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:48.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:47.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:48.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:47.658+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:48.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:48.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:48.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:48.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:48.170+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:49.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:48.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:48.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:48.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:48.698+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:48.854+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:48.854+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:48.854+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:49.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:48.855+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:49.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:49.218+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:49.218+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:49.218+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:49.219+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:50.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:49.709+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:49.709+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:49.709+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:49.709+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:49.856+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:49.856+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:49.856+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:50.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:49.856+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:50.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:50.205+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:50.205+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:50.205+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:50.206+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:51.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:50.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:50.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:50.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:50.821+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:51.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:50.746+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:51.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:50.746+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:51.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:50.746+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:51.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:50.747+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:51.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:51.205+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:51.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:51.205+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:51.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:51.205+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:51.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:51.206+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:52.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:51.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:51.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:51.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:51.822+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:52.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:51.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:52.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:51.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:52.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:51.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:52.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:51.792+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:52.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:09:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:09:52.035+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16599 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:09:52.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:52.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:52.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:52.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:52.163+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:53.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:52.778+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:52.778+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:52.778+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:52.779+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:53.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:52.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:53.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:52.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:53.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:52.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:53.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:52.787+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:53.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:53.173+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:53.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:53.173+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:53.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:53.173+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:53.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:53.174+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:54.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:53.819+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:54.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:53.819+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:54.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:53.819+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:54.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:53.819+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:53.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:53.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:53.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:54.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:53.821+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:54.207+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:54.207+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:54.207+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:54.208+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:55.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:54.782+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:55.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:54.782+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:55.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:54.782+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:55.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:54.783+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:55.085 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:54.813+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:55.085 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:54.814+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:55.085 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:54.814+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:55.085 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:54.814+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:55.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:55.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:55.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:55.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:55.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:55.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:55.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:55.197+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:56.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:55.777+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:55.777+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:55.777+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:55.778+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:55.780+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:55.780+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:55.780+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:56.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:55.781+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:56.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:56.206+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:56.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:56.206+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:56.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:56.206+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:56.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:56.207+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:57.040 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:56.779+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:57.040 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:56.779+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:57.040 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:56.779+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:57.041 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:56.780+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:57.041 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:56.788+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:57.041 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:56.788+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:57.041 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:56.788+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:57.041 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:56.789+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:57.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:09:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:09:57.038+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16656 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:09:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:57.174+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:57.174+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:57.174+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:57.175+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:58.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:57.834+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:57.834+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:57.834+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:57.835+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:57.794+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:57.794+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:57.794+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:58.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:57.794+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:58.133+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:58.133+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:58.133+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:58.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:58.133+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:59.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:58.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:58.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:58.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:09:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:58.841+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:58.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:58.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:58.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:59.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:58.824+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:09:59.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:59.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:59.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:59.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:59.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:59.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:09:59.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:09:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:09:59.104+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:09:59.859 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:59.803+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:59.859 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:59.803+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:59.859 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:59.803+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:09:59.859 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:09:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:09:59.803+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:00.070+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:00.070+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:00.070+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:00.071+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:59.857+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:59.857+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:59.857+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:00.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:09:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:09:59.858+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:01.057 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:00.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:01.058 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:00.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:01.058 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:00.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:01.058 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:00.824+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:01.058 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:00.774+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:01.058 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:00.774+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:01.058 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:00.774+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:01.059 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:00.774+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:01.056+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:01.056+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:01.056+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:01.056+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:02.041 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:01.794+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:02.042 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:01.794+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:02.042 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:01.794+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:02.042 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:01.794+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:02.042 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:01.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:02.042 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:01.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:02.042 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:01.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:02.042 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:01.736+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:02.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:10:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:10:02.040+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16715 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:10:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:02.041+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:02.041+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:02.041+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:02.042+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:03.052 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:02.825+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:03.052 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:02.825+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:03.053 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:02.825+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:03.053 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:02.826+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:03.053 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:02.741+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:03.053 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:02.741+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:03.053 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:02.741+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:03.053 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:02.742+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:03.050+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:03.050+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:03.050+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:03.051+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:04.031 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:03.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:04.031 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:03.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:04.031 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:03.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:04.031 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:03.817+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:04.032 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:03.715+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:04.032 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:03.715+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:04.032 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:03.715+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:04.032 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:03.716+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:04.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:04.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:04.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:04.030+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:05.013 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:04.836+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:05.013 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:04.836+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:05.013 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:04.836+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:05.013 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:04.836+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:05.014 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:04.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:05.014 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:04.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:05.014 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:04.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:05.014 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:04.699+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:05.011+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:05.011+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:05.011+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:05.012+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:06.048 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:05.861+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:06.049 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:05.861+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:06.049 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:05.861+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:06.049 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:05.862+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:06.049 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:05.706+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:06.049 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:05.706+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:06.050 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:05.706+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:06.050 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:05.707+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:06.052+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:06.052+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:06.052+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:06.052+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:07.043 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:06.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:07.043 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:06.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:07.044 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:06.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:07.044 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:06.873+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:07.044 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:06.707+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:07.044 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:06.707+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:07.044 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:06.707+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:07.045 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:06.708+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:07.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:07.047+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:07.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:07.047+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:07.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:07.047+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:07.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:07.048+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:07.332 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:10:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:10:07.041+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16773 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:10:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:07.863+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:07.863+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:07.863+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:07.864+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:08.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:07.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:08.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:07.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:08.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:07.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:08.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:07.734+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:08.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:08.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:08.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:08.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:08.097+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:09.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:08.882+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:08.882+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:08.882+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:08.883+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:08.761+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:08.761+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:08.761+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:08.762+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:09.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:09.098+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:09.098+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:09.098+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:09.098+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:09.893+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:09.893+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:09.893+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:09.893+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:09.764+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:09.764+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:10.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:09.764+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:10.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:09.764+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:10.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:10.125+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:10.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:10.125+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:10.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:10.125+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:10.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:10.126+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:11.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:10.804+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:10.804+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:10.804+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:10.804+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:10.881+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:10.881+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:11.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:10.881+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:11.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:10.882+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:11.540 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:11.138+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:11.541 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:11.138+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:11.541 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:11.138+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:11.541 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:11.138+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:11.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:11.772+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:11.772+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:11.772+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:11.773+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:12.168 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:10:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:10:12.043+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16831 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:10:12.168 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:11.898+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:12.168 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:11.898+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:12.169 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:11.898+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:12.169 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:11.898+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:12.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:12.166+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:12.166+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:12.166+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:12.167+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:13.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:12.942+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:12.942+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:12.942+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:12.942+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:13.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:12.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:13.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:12.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:13.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:12.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:13.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:12.773+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:13.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:13.214+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:13.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:13.214+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:13.214+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:13.215+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:13.895 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:13.769+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:13.895 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:13.769+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:13.895 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:13.769+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:13.896 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:13.770+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:14.247 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:13.893+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:14.248 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:13.893+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:14.248 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:13.893+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:14.248 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:13.894+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:14.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:14.246+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:14.246+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:14.246+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:14.247+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:14.861 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:14.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:14.861 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:14.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:14.861 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:14.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:14.861 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:14.788+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:15.264 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:14.859+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:15.265 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:14.859+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:15.265 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:14.859+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:15.265 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:14.860+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:15.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:15.279+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:15.279+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:15.279+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:15.280+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:16.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:15.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:15.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:15.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:15.875+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:16.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:15.816+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:16.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:15.816+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:15.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:15.817+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:16.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:16.272+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:16.272+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:16.272+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:16.272+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:16.911 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:16.816+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:16.911 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:16.816+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:16.911 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:16.816+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:16.911 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:16.817+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:17.294 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:16.909+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:17.294 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:16.909+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:17.294 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:16.909+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:17.295 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:16.910+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:17.295 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:10:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:10:17.045+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16888 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:10:17.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:17.292+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:17.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:17.292+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:17.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:17.292+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:17.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:17.293+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:18.310 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:17.938+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:18.311 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:17.938+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:18.311 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:17.938+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:18.311 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:17.939+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:18.311 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:17.858+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:18.311 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:17.858+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:18.311 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:17.858+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:18.312 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:17.859+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:18.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:18.309+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:18.309+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:18.309+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:18.309+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:19.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:18.912+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:19.358 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:18.912+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:19.359 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:18.912+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:19.359 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:18.912+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:19.359 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:18.882+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:19.360 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:18.882+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:19.360 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:18.882+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:19.360 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:18.883+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:19.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:19.342+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:19.342+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:19.342+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:19.343+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:20.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:19.837+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:19.837+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:19.837+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:19.838+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:19.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:19.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:19.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:20.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:19.933+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:20.791 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:20.384+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:20.792 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:20.384+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:20.792 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:20.384+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:20.792 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:20.385+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:20.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:20.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:20.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:20.909+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:21.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:20.790+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:21.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:20.790+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:21.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:20.790+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:21.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:20.791+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:21.714 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:21.429+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:21.714 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:21.429+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:21.715 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:21.429+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:21.715 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:21.430+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:22.048 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:21.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:22.049 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:21.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:22.049 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:21.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:22.049 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:21.862+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:22.049 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:21.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:22.049 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:21.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:22.049 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:21.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:22.050 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:21.749+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:22.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:10:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:10:22.047+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 16948 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:10:22.776 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:22.432+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:22.776 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:22.432+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:22.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:22.432+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:22.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:22.433+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:22.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:22.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:22.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:22.903+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:22.774+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:22.774+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:22.774+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:23.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:22.775+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:23.816 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:23.461+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:23.816 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:23.461+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:23.816 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:23.461+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:23.816 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:23.462+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:24.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:23.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:23.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:23.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:23.916+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:23.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:23.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:24.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:23.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:24.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:23.815+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:24.773 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:24.447+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:24.773 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:24.447+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:24.773 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:24.447+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:24.773 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:24.448+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:25.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:24.771+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:25.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:24.771+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:25.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:24.771+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:25.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:24.772+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:24.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:24.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:24.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:24.872+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:25.762 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:25.430+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:25.763 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:25.430+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:25.763 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:25.430+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:25.763 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:25.430+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:26.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:25.854+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:25.854+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:25.854+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:25.855+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:25.760+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:25.760+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:25.760+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:26.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:25.760+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:26.757 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:26.382+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:26.757 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:26.382+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:26.757 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:26.382+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:26.757 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:26.382+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:27.050 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:26.881+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:27.050 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:26.881+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:27.051 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:26.881+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:27.051 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:26.881+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:27.051 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:26.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:27.051 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:26.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:27.051 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:26.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:27.051 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:26.755+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:27.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:10:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:10:27.047+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 17006 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:10:27.723 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:27.432+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:27.724 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:27.432+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:27.724 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:27.432+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:27.724 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:27.433+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:28.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:27.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:27.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:27.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:27.863+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:28.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:27.721+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:28.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:27.721+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:28.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:27.721+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:28.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:27.721+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:28.750 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:28.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:28.750 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:28.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:28.750 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:28.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:28.751 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:28.441+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:29.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:28.747+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:28.747+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:28.747+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:28.748+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:29.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:28.849+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:29.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:28.849+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:29.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:28.849+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:29.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:28.850+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:29.793 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:29.425+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:29.794 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:29.425+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:29.794 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:29.425+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:29.794 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:29.426+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:29.807+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:29.807+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:29.807+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:29.807+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:29.791+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:29.791+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:29.791+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:30.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:29.791+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:30.762 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:30.430+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:30.762 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:30.430+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:30.762 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:30.430+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:30.762 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:30.431+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:30.826+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:30.826+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:30.826+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:30.826+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:30.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:30.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:30.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:30.760+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:31.728 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:31.397+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:31.728 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:31.397+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:31.729 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:31.397+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:31.729 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:31.398+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:32.052 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:31.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:32.052 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:31.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:32.052 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:31.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:32.052 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:31.800+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:32.053 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:31.727+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:32.053 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:31.727+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:32.053 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:31.727+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:32.053 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:31.727+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:32.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:10:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:10:32.050+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 17065 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:10:32.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:32.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:32.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:32.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:32.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:32.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:32.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:32.424+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:32.762+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:32.762+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:32.762+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:32.762+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:32.747+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:33.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:32.747+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:33.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:32.747+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:33.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:32.748+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:33.696 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:33.423+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:33.697 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:33.423+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:33.697 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:33.423+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:33.697 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:33.424+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:34.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:33.718+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:33.718+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:33.718+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:33.718+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:33.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:33.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:33.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:34.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:33.746+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:34.768 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:34.694+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:34.768 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:34.694+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:34.768 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:34.694+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:34.768 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:34.694+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:34.769 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:34.473+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:34.769 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:34.473+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:34.769 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:34.473+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:34.769 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:34.474+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:35.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:34.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:34.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:34.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:34.767+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:35.740 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:35.460+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:35.740 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:35.460+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:35.740 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:35.460+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:35.740 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:35.460+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:36.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:35.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:35.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:35.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:35.748+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:35.738+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:35.738+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:35.738+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:36.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:35.739+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:36.721 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:36.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:36.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:36.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:36.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:36.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:36.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:36.414+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:37.000 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:36.735+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:37.000 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:36.735+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:37.000 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:36.735+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:37.000 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:36.736+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:37.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:36.720+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:37.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:36.720+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:37.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:36.720+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:37.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:36.721+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:37.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:10:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:10:37.052+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 17123 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:10:37.680 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:37.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:37.680 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:37.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:37.681 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:37.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:37.681 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:37.428+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:37.767+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:37.767+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:37.767+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:37.767+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:37.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:37.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:38.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:37.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:38.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:37.679+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:38.793 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:38.673+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:38.793 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:38.673+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:38.794 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:38.673+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:38.794 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:38.673+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:38.794 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:38.461+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:38.794 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:38.461+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:38.794 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:38.461+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:38.794 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:38.461+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:39.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:38.792+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:39.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:38.792+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:38.792+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:38.792+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:39.787 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:39.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:39.787 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:39.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:39.787 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:39.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:39.787 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:39.648+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:39.788 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:39.454+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:39.788 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:39.454+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:39.788 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:39.454+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:39.788 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:39.454+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:40.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:39.785+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:40.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:39.785+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:39.785+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:39.786+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:40.761 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:40.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:40.761 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:40.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:40.761 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:40.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:40.761 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:40.608+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:40.761 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:40.415+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:40.762 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:40.415+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:40.762 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:40.415+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:40.762 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:40.416+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:40.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:40.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:40.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:40.760+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:41.811 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:41.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:41.811 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:41.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:41.811 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:41.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:41.811 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:41.603+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:41.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:41.454+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:41.812 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:41.454+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:41.812 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:41.454+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:41.812 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:41.455+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:42.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:10:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:10:42.053+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 17181 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:10:42.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:41.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:42.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:41.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:42.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:41.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:42.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:41.810+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:42.795 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:42.419+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:42.795 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:42.419+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:42.796 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:42.419+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:42.796 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:42.420+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:42.796 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:42.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:42.796 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:42.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:42.796 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:42.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:42.796 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:42.624+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:43.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:42.793+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:43.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:42.793+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:42.793+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:42.794+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:43.626 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:43.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:43.626 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:43.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:43.626 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:43.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:43.627 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:43.603+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:43.627 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:43.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:43.627 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:43.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:43.627 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:43.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:43.627 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:43.375+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:44.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:43.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:43.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:43.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:43.777+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:44.751 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:44.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:44.752 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:44.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:44.752 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:44.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:44.752 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:44.593+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:44.752 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:44.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:44.752 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:44.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:44.752 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:44.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:44.753 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:44.383+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:45.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:44.750+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:45.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:44.750+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:44.750+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:44.750+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:45.800 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:45.381+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:45.801 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:45.381+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:45.801 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:45.381+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:45.801 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:45.381+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:45.801 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:45.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:45.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:45.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:45.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:45.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:45.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:45.545+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:46.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:45.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:45.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:45.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:45.800+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:46.689 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:46.430+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:46.690 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:46.430+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:46.690 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:46.430+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:46.690 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:46.431+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:46.690 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:46.570+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:46.690 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:46.570+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:46.691 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:46.570+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:46.691 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:46.570+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:47.057 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:46.802+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:47.058 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:46.802+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:47.058 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:46.802+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:47.058 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:46.802+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:47.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:10:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:10:47.055+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 17240 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:10:47.829 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:47.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:47.829 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:47.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:47.829 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:47.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:47.829 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:47.594+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:47.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:47.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:47.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:47.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:47.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:47.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:47.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:47.394+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:48.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:47.827+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:47.827+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:47.827+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:47.828+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:48.429+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:48.429+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:48.429+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:48.430+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:48.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:48.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:48.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:48.558+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:48.873+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:48.873+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:48.873+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:48.874+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:49.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:49.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:49.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:49.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:49.437+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:49.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:49.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:49.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:49.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:49.569+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:50.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:49.891+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:49.891+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:49.891+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:49.891+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:50.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:50.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:50.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:50.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:50.401+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:50.536+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:50.536+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:50.536+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:50.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:50.537+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:50.916+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:50.916+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:50.916+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:50.917+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:51.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:51.398+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:51.398+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:51.398+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:51.399+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:51.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:51.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:51.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:51.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:51.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:51.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:51.578+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:52.270 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:10:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:10:52.057+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 17299 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:10:52.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:51.931+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:52.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:51.931+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:52.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:51.931+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:52.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:51.931+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:52.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:52.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:52.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:52.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:52.353+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:52.928 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:52.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:52.928 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:52.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:52.928 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:52.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:52.928 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:52.590+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:53.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:52.926+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:52.926+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:52.926+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:52.927+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:53.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:53.393+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:53.393+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:53.393+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:53.394+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:53.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:53.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:53.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:53.587+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:53.976+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:53.976+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:53.976+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:53.976+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:54.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:54.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:54.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:54.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:54.569+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:54.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:54.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:54.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:54.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:54.389+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:55.200 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:54.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:55.201 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:54.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:55.201 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:54.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:55.201 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:54.936+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:55.411 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:55.408+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:55.411 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:55.408+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:55.412 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:55.408+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:55.560 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:55.409+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:55.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:55.558+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:55.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:55.558+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:55.558+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:55.559+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:55.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:55.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:55.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:55.903+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:56.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:56.433+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:56.433+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:56.433+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:56.433+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:56.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:56.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:56.557+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:56.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:56.558+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:57.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:10:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:10:57.059+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 17356 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:10:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:56.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:56.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:56.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:56.867+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:57.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:57.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:57.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:57.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:57.400+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:57.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:57.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:57.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:57.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:57.554+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:57.838+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:57.838+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:57.838+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:57.838+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:58.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:58.444+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:58.444+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:58.444+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:58.445+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:58.506+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:58.506+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:58.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:58.506+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:58.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:58.507+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:59.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:58.832+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:58.832+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:58.832+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:10:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:58.833+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:10:59.809 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:59.538+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:59.809 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:59.538+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:59.809 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:59.538+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:10:59.809 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:10:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:10:59.539+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:10:59.810 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:59.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:59.810 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:59.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:59.810 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:59.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:10:59.810 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:10:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:10:59.413+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:00.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:59.807+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:00.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:59.807+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:00.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:59.807+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:00.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:10:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:10:59.808+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:00.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:00.406+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:00.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:00.406+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:00.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:00.406+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:00.824 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:00.407+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:00.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:00.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:00.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:00.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:00.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:00.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:00.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:00.495+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:00.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:00.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:00.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:00.822+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:01.825 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:01.396+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:01.826 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:01.396+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:01.826 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:01.396+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:01.826 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:01.397+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:01.826 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:01.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:01.827 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:01.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:01.827 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:01.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:01.827 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:01.483+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:02.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:11:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:11:02.061+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 17415 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:11:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:01.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:01.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:01.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:02.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:01.824+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:02.807 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:02.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:02.808 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:02.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:02.808 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:02.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:02.808 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:02.479+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:02.808 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:02.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:02.809 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:02.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:02.809 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:02.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:02.809 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:02.354+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:02.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:02.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:02.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:02.806+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:03.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:03.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:03.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:03.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:03.779 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:03.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:03.779 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:03.446+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:03.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:03.376+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:03.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:03.376+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:03.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:03.376+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:03.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:03.376+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:04.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:03.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:03.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:03.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:03.776+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:04.798 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:04.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:04.799 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:04.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:04.799 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:04.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:04.799 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:04.427+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:04.799 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:04.439+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:04.800 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:04.439+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:04.800 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:04.439+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:04.800 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:04.440+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:05.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:04.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:04.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:04.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:04.796+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:05.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:05.432+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:05.432+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:05.432+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:05.433+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:05.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:05.412+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:05.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:05.412+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:05.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:05.412+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:05.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:05.412+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:06.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:05.830+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:05.830+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:05.830+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:05.830+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:06.822 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:06.477+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:06.822 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:06.477+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:06.822 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:06.477+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:06.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:06.478+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:06.823 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:06.423+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:06.823 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:06.423+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:06.823 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:06.423+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:06.823 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:06.423+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:07.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:11:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:11:07.061+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 17474 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:11:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:06.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:06.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:06.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:06.820+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:07.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:07.439+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:07.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:07.439+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:07.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:07.439+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:07.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:07.439+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:07.803 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:07.491+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:07.803 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:07.491+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:07.803 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:07.491+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:07.803 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:07.492+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:07.800+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:07.800+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:07.800+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:07.800+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:08.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:08.488+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:08.488+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:08.488+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:08.489+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:08.400+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:08.400+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:08.400+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:08.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:08.400+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:09.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:08.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:08.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:08.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:08.842+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:09.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:09.480+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:09.480+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:09.480+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:09.481+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:09.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:09.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:09.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:09.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:09.399+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:09.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:09.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:09.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:09.884+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:10.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:10.441+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:10.441+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:10.441+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:10.441+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:10.372+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:10.372+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:10.372+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:10.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:10.372+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:11.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:10.863+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:10.863+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:10.863+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:10.863+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:11.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:11.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:11.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:11.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:11.417+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:11.412+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:11.412+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:11.412+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:11.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:11.412+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:12.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:11:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:11:12.064+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 17533 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:11:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:11.911+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:11.911+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:11.911+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:11.911+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:12.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:12.425+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:12.425+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:12.425+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:12.425+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:12.405+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:12.405+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:12.405+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:12.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:12.405+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:13.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:12.925+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:12.925+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:12.925+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:12.925+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:13.628 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:13.376+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:13.629 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:13.376+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:13.629 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:13.376+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:13.629 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:13.376+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:13.629 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:13.444+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:13.629 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:13.444+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:13.629 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:13.444+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:13.630 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:13.445+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:14.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:13.912+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:13.912+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:13.912+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:13.913+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:14.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:14.480+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:14.480+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:14.480+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:14.480+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:14.425+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:14.425+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:14.425+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:14.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:14.425+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:14.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:14.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:14.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:14.876+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:15.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:15.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:15.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:15.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:15.507+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:15.449+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:15.449+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:15.449+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:15.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:15.450+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:16.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:15.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:15.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:15.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:15.921+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:16.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:16.534+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:16.534+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:16.534+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:16.535+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:16.492+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:16.492+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:16.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:16.492+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:16.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:16.492+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:17.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:11:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:11:17.066+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 17591 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:11:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:16.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:16.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:16.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:16.961+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:17.697 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:17.472+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:17.697 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:17.472+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:17.697 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:17.472+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:17.698 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:17.473+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:17.698 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:17.542+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:17.698 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:17.542+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:17.698 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:17.542+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:17.698 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:17.542+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:17.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:17.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:17.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:17.990+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:18.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:18.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:18.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:18.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:18.556+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:18.430+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:18.430+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:18.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:18.430+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:18.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:18.430+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:19.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:19.030+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:19.030+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:19.030+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:19.030+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:19.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:19.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:19.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:19.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:19.778 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:19.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:19.778 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:19.562+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:19.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:19.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:19.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:19.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:19.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:19.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:19.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:19.441+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:20.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:20.022+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:20.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:20.022+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:20.022+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:20.023+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:20.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:20.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:20.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:20.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:20.588+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:20.450+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:20.450+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:20.450+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:20.450+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:21.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:21.054+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:21.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:21.054+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:21.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:21.054+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:21.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:21.054+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:21.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:21.570+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:21.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:21.570+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:21.570+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:21.571+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:21.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:21.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:21.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:21.489+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:22.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:11:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:11:22.068+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 17650 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:11:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:22.010+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:22.010+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:22.010+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:22.011+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:22.819 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:22.601+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:22.819 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:22.601+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:22.820 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:22.601+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:22.820 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:22.601+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:22.820 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:22.462+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:22.820 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:22.462+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:22.820 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:22.462+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:22.820 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:22.462+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:23.055+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:23.055+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:23.055+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:23.056+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:23.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:23.564+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:23.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:23.564+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:23.564+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:23.564+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:23.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:23.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:23.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:23.476+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:24.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:24.076+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:24.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:24.076+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:24.076+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:24.077+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:24.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:24.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:24.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:24.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:24.525+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:24.521+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:24.521+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:24.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:24.521+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:24.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:24.521+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:25.514 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:25.087+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:25.514 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:25.087+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:25.514 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:25.087+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:25.514 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:25.087+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:25.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:25.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:25.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:25.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:25.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:25.551+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:25.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:25.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:25.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:25.513+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:26.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:26.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:26.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:26.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:26.088+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:26.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:26.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:26.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:26.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:26.546+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:26.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:26.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:26.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:26.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:26.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:26.506+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:27.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:11:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:11:27.069+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 17709 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:11:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:27.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:27.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:27.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:27.121+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:27.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:27.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:27.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:27.501+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:27.501+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:27.501+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:27.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:27.501+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:27.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:27.502+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:28.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:28.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:28.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:28.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:28.107+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:28.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:28.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:28.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:28.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:28.491+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:28.539+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:28.539+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:28.539+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:28.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:28.540+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:29.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:29.128+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:29.128+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:29.128+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:29.129+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:29.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:29.531+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:29.531+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:29.531+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:29.532+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:29.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:29.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:29.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:29.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:29.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:29.590+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:30.538 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:30.099+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:30.538 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:30.099+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:30.538 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:30.099+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:30.539 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:30.099+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:30.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:30.537+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:30.537+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:30.537+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:30.537+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:30.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:30.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:30.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:30.629+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:31.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:31.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:31.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:31.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:31.075+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:31.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:31.673+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:31.673+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:31.673+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:31.674+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:31.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:31.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:31.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:31.528+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:32.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:11:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:11:32.071+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 17768 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:11:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:32.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:32.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:32.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:32.099+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:32.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:32.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:32.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:32.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:32.626+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:32.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:32.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:32.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:32.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:32.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:32.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:32.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:32.524+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:33.572 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:33.135+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:33.576 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:33.135+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:33.576 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:33.135+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:33.577 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:33.135+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:33.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:33.571+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:33.571+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:33.571+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:33.572+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:33.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:33.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:33.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:33.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:33.616+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:34.551 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:34.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:34.551 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:34.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:34.552 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:34.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:34.552 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:34.123+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:34.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:34.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:34.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:34.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:34.550+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:34.573+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:34.573+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:34.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:34.573+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:34.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:34.573+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:35.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:35.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:35.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:35.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:35.074+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:35.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:35.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:35.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:35.585+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:35.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:35.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:35.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:35.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:35.546+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:36.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:36.039+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:36.039+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:36.039+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:36.039+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:36.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:36.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:36.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:36.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:36.543+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:36.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:36.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:36.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:36.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:36.583+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:37.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:11:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:11:37.073+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 17826 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:11:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:37.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:37.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:37.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:37.008+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:37.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:37.541+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:37.541+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:37.541+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:37.542+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:37.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:37.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:37.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:37.586+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:38.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:37.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:37.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:37.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:37.967+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:38.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:38.547+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:38.547+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:38.547+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:38.548+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:38.574+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:38.574+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:38.574+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:38.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:38.575+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:38.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:38.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:38.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:38.934+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:39.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:39.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:39.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:39.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:39.528+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:39.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:39.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:39.552+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:39.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:39.553+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:40.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:39.901+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:39.901+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:39.901+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:39.902+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:40.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:40.497+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:40.497+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:40.497+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:40.498+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:40.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:40.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:40.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:40.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:40.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:40.582+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:41.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:40.887+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:40.887+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:40.887+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:40.888+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:41.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:41.489+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:41.489+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:41.489+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:41.490+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:41.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:41.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:41.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:41.555+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:42.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:11:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:11:42.074+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 17885 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:11:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:41.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:41.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:41.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:41.843+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:42.813 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:42.476+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:42.813 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:42.476+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:42.813 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:42.476+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:42.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:42.476+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:42.814 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:42.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:42.814 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:42.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:42.814 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:42.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:42.814 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:42.546+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:43.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:42.810+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:42.810+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:42.810+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:42.811+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:43.798 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:43.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:43.799 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:43.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:43.799 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:43.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:43.799 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:43.437+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:43.799 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:43.518+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:43.799 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:43.518+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:43.800 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:43.518+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:43.800 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:43.518+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:44.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:43.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:44.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:43.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:43.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:43.796+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:44.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:44.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:44.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:44.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:44.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:44.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:44.779 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:44.478+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:44.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:44.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:44.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:44.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:44.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:44.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:44.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:44.403+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:45.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:44.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:44.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:44.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:44.776+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:45.812 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:45.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:45.813 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:45.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:45.813 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:45.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:45.813 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:45.480+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:45.813 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:45.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:45.813 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:45.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:45.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:45.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:45.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:45.392+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:46.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:45.810+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:45.810+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:45.810+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:45.812+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:46.813 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:46.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:46.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:46.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:46.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:46.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:46.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:46.441+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:46.814 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:46.465+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:46.814 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:46.465+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:46.814 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:46.465+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:46.815 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:46.465+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:47.078 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:46.812+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:47.078 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:46.812+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:47.078 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:46.812+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:47.079 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:46.812+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:47.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:11:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:11:47.076+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 17944 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:11:47.825 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:47.470+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:47.825 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:47.470+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:47.826 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:47.470+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:47.826 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:47.471+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:47.826 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:47.424+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:47.826 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:47.424+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:47.826 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:47.424+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:47.826 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:47.425+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:48.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:47.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:47.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:47.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:47.824+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:48.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:48.433+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:48.433+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:48.433+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:48.434+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:48.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:48.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:48.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:48.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:48.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:48.404+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:49.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:48.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:49.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:48.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:48.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:48.777+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:49.788 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:49.446+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:49.788 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:49.446+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:49.788 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:49.446+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:49.788 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:49.446+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:49.788 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:49.392+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:49.789 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:49.392+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:49.789 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:49.392+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:49.789 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:49.392+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:50.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:49.786+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:49.786+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:49.786+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:49.787+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:50.822 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:50.419+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:50.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:50.419+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:50.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:50.419+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:50.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:50.420+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:50.823 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:50.374+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:50.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:50.374+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:50.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:50.374+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:50.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:50.375+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:51.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:50.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:50.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:50.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:50.822+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:51.821 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:51.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:51.822 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:51.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:51.822 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:51.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:51.822 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:51.342+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:51.822 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:51.464+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:51.822 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:51.464+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:51.822 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:51.464+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:51.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:51.465+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:52.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:51.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:52.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:51.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:52.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:51.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:52.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:51.820+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:52.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:11:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:11:52.078+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 18003 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:11:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:52.324+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:52.324+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:52.324+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:52.325+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:52.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:52.433+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:52.433+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:52.433+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:52.434+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:53.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:52.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:52.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:52.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:52.838+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:53.633 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:53.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:53.633 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:53.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:53.633 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:53.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:53.633 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:53.363+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:53.634 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:53.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:53.634 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:53.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:53.634 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:53.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:53.634 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:53.411+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:54.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:53.806+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:53.806+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:53.806+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:53.807+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:54.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:54.369+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:54.369+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:54.369+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:54.370+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:54.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:54.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:54.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:54.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:54.428+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:54.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:54.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:54.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:54.830+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:55.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:55.333+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:55.333+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:55.333+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:55.334+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:55.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:55.403+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:55.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:55.403+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:55.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:55.403+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:55.404+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:55.852+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:55.852+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:55.852+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:55.852+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:56.640 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:56.365+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:56.641 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:56.365+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:56.641 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:56.365+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:56.641 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:56.366+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:56.641 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:56.425+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:56.641 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:56.425+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:56.642 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:56.425+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:56.642 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:56.425+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:57.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:56.847+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:56.847+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:56.847+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:56.848+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:57.373 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:11:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:11:57.080+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 18061 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:11:57.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:57.441+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:57.441+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:57.441+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:57.442+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:57.372+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:57.372+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:57.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:57.372+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:57.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:57.372+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:57.864+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:57.864+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:57.864+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:57.864+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:58.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:58.386+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:58.386+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:58.386+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:58.387+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:58.488+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:58.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:58.488+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:58.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:58.488+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:58.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:58.489+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:58.841+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:58.841+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:58.841+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:11:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:58.842+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:11:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:59.433+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:59.433+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:59.433+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:11:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:11:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:11:59.434+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:11:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:59.465+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:59.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:59.465+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:59.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:59.465+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:11:59.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:11:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:11:59.465+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:00.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:59.870+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:59.870+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:59.870+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:11:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:11:59.872+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:00.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:00.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:00.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:00.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:00.464+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:00.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:00.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:00.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:00.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:00.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:00.427+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:01.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:00.868+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:00.868+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:00.868+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:00.869+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:01.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:01.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:01.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:01.421+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:01.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:01.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:01.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:01.419+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:02.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:12:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:12:02.082+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 18120 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:12:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:01.857+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:01.857+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:01.857+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:01.858+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:02.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:02.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:02.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:02.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:02.404+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:02.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:02.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:02.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:02.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:02.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:02.420+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:03.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:02.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:02.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:02.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:02.905+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:03.631 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:03.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:03.631 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:03.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:03.632 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:03.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:03.632 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:03.447+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:03.632 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:03.363+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:03.632 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:03.363+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:03.632 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:03.363+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:03.633 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:03.364+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:03.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:03.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:03.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:03.909+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:04.598 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:04.439+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:04.598 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:04.439+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:04.598 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:04.439+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:04.598 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:04.440+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:04.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:04.341+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:04.599 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:04.341+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:04.599 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:04.341+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:04.599 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:04.342+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:05.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:04.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:05.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:04.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:05.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:04.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:05.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:04.873+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:05.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:05.295+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:05.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:05.295+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:05.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:05.295+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:05.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:05.296+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:05.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:05.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:05.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:05.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:05.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:05.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:05.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:05.486+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:06.275 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:05.865+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:06.275 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:05.865+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:06.276 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:05.865+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:06.276 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:05.866+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:06.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:06.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:06.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:06.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:06.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:06.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:06.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:06.274+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:06.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:06.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:06.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:06.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:06.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:06.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:06.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:06.479+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:07.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:06.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:06.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:06.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:06.824+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:07.451 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:12:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:12:07.084+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 18179 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:12:07.451 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:07.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:07.452 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:07.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:07.452 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:07.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:07.452 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:07.243+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:07.779 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:07.449+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:07.779 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:07.449+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:07.779 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:07.449+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:07.779 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:07.450+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:07.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:07.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:07.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:07.778+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:08.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:08.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:08.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:08.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:08.470+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:08.224+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:08.224+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:08.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:08.224+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:08.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:08.225+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:08.929 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:08.763+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:08.929 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:08.763+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:08.930 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:08.763+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:08.930 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:08.764+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:09.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:09.240+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:09.240+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:09.240+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:09.241+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:09.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:09.425+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:09.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:09.425+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:09.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:09.425+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:09.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:09.425+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:09.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:09.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:09.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:09.776+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:10.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:10.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:10.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:10.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:10.455+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:10.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:10.218+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:10.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:10.218+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:10.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:10.218+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:10.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:10.219+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:11.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:10.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:10.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:10.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:10.755+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:11.486 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:11.214+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:11.486 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:11.214+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:11.487 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:11.214+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:11.487 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:11.215+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:11.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:11.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:11.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:11.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:11.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:11.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:11.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:11.486+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:12.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:11.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:12.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:11.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:12.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:11.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:12.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:11.775+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:12.444 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:12:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:12:12.086+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 18238 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:12:12.444 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:12.212+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:12.445 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:12.212+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:12.445 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:12.212+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:12.445 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:12.213+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:12.792 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:12.442+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:12.792 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:12.442+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:12.792 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:12.442+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:12.793 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:12.443+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:13.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:12.790+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:13.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:12.790+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:12.790+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:12.791+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:13.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:13.234+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:13.234+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:13.234+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:13.235+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:13.452+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:13.452+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:13.452+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:13.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:13.453+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:14.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:13.795+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:13.795+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:13.795+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:13.796+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:14.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:14.252+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:14.252+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:14.252+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:14.253+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:14.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:14.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:14.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:14.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:14.481+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:15.072 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:14.793+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:15.073 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:14.793+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:15.073 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:14.793+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:15.073 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:14.794+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:15.251+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:15.251+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:15.251+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:15.251+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:15.785 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:15.506+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:15.785 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:15.506+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:15.786 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:15.506+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:15.786 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:15.507+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:16.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:15.784+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:15.784+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:15.784+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:15.785+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:16.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:16.278+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:16.278+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:16.278+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:16.279+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:16.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:16.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:16.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:16.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:16.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:16.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:16.471+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:17.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:16.741+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:16.741+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:16.741+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:16.742+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:17.501 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:12:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:12:17.087+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 18296 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:12:17.501 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:17.298+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:17.501 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:17.298+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:17.501 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:17.298+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:17.502 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:17.298+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:17.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:17.498+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:17.498+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:17.498+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:17.499+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:17.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:17.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:17.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:17.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:17.726+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:18.525 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:18.249+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:18.526 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:18.249+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:18.526 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:18.249+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:18.526 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:18.250+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:18.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:18.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:18.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:18.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:18.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:18.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:18.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:18.524+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:18.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:18.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:18.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:18.775+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:19.533 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:19.258+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:19.533 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:19.258+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:19.534 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:19.258+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:19.534 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:19.258+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:19.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:19.530+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:19.530+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:19.530+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:19.531+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:19.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:19.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:19.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:19.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:19.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:19.730+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:20.571 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:20.295+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:20.571 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:20.295+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:20.571 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:20.295+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:20.571 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:20.295+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:20.699+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:20.699+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:20.699+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:20.699+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:20.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:20.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:20.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:20.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:20.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:20.569+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:21.525 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:21.313+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:21.525 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:21.313+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:21.526 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:21.313+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:21.526 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:21.314+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:21.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:21.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:21.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:21.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:21.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:21.523+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:21.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:21.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:21.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:21.696+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:22.532 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:12:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:12:22.088+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 18354 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:12:22.532 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:22.321+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:22.532 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:22.321+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:22.533 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:22.321+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:22.533 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:22.321+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:22.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:22.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:22.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:22.718+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:22.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:22.530+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:22.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:22.530+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:22.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:22.530+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:22.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:22.530+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:23.633 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:23.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:23.633 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:23.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:23.633 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:23.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:23.633 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:23.557+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:23.634 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:23.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:23.634 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:23.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:23.634 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:23.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:23.634 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:23.371+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:24.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:23.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:23.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:23.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:23.766+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:24.547 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:24.357+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:24.548 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:24.357+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:24.548 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:24.357+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:24.548 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:24.358+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:24.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:24.785+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:24.785+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:24.785+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:24.786+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:24.546+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:24.546+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:24.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:24.546+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:24.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:24.547+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:25.579 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:25.309+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:25.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:25.309+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:25.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:25.309+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:25.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:25.310+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:25.748+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:25.748+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:25.748+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:25.748+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:25.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:25.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:25.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:25.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:25.578+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:26.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:26.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:26.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:26.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:26.328+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:27.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:26.707+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:26.707+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:26.707+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:26.708+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:26.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:26.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:26.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:27.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:26.587+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:27.361 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:12:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:12:27.091+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 18413 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:12:27.361 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:27.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:27.362 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:27.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:27.362 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:27.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:27.362 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:27.284+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:28.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:27.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:27.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:27.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:27.730+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:28.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:27.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:28.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:27.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:28.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:27.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:28.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:27.593+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:28.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:28.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:28.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:28.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:28.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:28.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:28.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:28.299+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:28.965 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:28.694+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:28.965 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:28.694+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:28.966 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:28.694+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:28.966 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:28.695+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:28.966 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:28.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:28.966 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:28.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:28.966 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:28.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:28.966 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:28.624+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:29.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:29.319+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:29.319+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:29.319+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:29.319+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:29.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:29.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:29.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:29.696+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:29.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:29.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:29.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:30.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:29.649+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:30.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:30.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:30.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:30.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:30.273+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:30.678+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:30.678+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:30.678+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:30.679+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:30.692+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:30.692+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:30.692+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:30.693+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:31.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:31.227+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:31.227+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:31.227+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:31.228+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:32.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:31.647+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:31.647+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:31.647+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:31.647+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:31.683+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:31.683+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:31.683+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:32.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:31.683+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:32.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:12:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:12:32.093+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 18472 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:12:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:32.206+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:32.206+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:32.206+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:32.207+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:32.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:32.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:32.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:32.639+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:32.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:32.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:32.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:33.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:32.704+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:33.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:33.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:33.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:33.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:33.224+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:34.058 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:33.643+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:34.058 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:33.643+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:34.058 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:33.643+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:34.058 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:33.644+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:34.059 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:33.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:34.059 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:33.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:34.059 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:33.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:34.059 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:33.755+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:34.222+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:34.222+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:34.222+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:34.223+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:35.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:34.669+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:34.669+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:34.669+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:34.669+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:34.781+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:34.781+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:35.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:34.781+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:35.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:34.782+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:35.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:35.237+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:35.237+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:35.237+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:35.238+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:36.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:35.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:35.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:35.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:35.683+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:35.804+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:35.804+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:35.804+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:36.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:35.804+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:36.576 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:36.232+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:36.577 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:36.232+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:36.577 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:36.232+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:36.577 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:36.233+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:36.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:36.642+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:36.642+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:36.642+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:36.642+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:37.191 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:12:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:12:37.095+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 18530 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:12:37.191 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:36.847+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:37.191 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:36.847+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:37.191 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:36.847+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:37.192 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:36.848+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:37.189+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:37.189+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:37.189+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:37.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:37.190+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:37.636+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:37.636+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:37.636+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:37.636+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:37.876+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:37.876+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:37.876+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:37.876+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:38.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:38.145+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:38.145+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:38.145+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:38.146+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:39.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:38.654+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:38.654+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:38.654+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:38.654+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:38.879+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:38.879+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:38.879+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:39.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:38.880+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:39.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:39.182+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:39.182+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:39.182+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:39.182+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:40.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:39.677+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:39.677+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:39.677+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:39.677+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:39.904+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:39.904+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:40.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:39.904+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:40.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:39.905+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:40.358 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T04:12:40.355+0000 7f6a468c8640 0 monclient(hunting): authenticate timed out after 300 2024-09-10T04:12:40.358 INFO:teuthology.orchestra.run.smithi112.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-09-10T04:12:40.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:40.167+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:40.167+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:40.167+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:40.167+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:40.937 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:40.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:40.938 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:40.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:40.938 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:40.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:40.938 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:40.651+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:41.200 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:41.158+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:41.201 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:41.158+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:41.201 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:41.158+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:41.201 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:41.159+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:41.201 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:40.936+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:41.202 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:40.936+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:41.202 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:40.936+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:41.202 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:40.937+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:41.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T04:12:41.204 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd blacklist ls 2024-09-10T04:12:41.493 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T04:12:41.962 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:41.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:41.962 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:41.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:41.962 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:41.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:41.962 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:41.627+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:42.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:12:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:12:42.097+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 18589 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:12:42.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:41.961+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:42.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:41.961+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:42.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:41.961+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:42.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:41.962+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:42.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:42.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:42.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:42.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:42.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:42.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:42.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:42.170+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:43.007 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:42.649+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:43.007 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:42.649+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:43.007 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:42.649+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:43.007 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:42.650+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:43.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:43.199+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:43.199+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:43.199+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:43.200+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:43.005+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:43.005+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:43.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:43.005+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:43.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:43.006+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:43.999 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:43.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:44.000 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:43.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:44.000 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:43.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:44.000 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:43.656+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:44.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:43.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:44.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:43.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:44.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:43.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:44.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:43.998+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:44.172+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:44.172+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:44.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:44.172+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:44.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:44.173+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:45.048 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:44.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:45.048 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:44.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:45.048 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:44.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:45.049 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:44.688+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:45.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:45.046+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:45.046+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:45.046+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:45.047+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:45.161+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:45.161+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:45.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:45.161+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:45.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:45.162+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:46.060 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:45.649+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:46.060 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:45.649+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:46.060 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:45.649+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:46.061 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:45.650+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:46.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:46.071+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:46.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:46.071+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:46.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:46.071+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:46.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:46.072+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:46.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:46.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:46.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:46.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:46.204+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:47.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:46.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:46.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:46.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:46.680+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:47.445 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:12:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:12:47.098+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 18648 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:12:47.445 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:47.104+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:47.446 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:47.104+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:47.446 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:47.104+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:47.446 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:47.106+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:47.446 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:47.217+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:47.446 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:47.217+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:47.446 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:47.217+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:47.447 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:47.218+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:48.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:47.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:47.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:47.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:47.699+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:48.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:48.264+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:48.264+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:48.264+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:48.265+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:48.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:48.103+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:48.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:48.103+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:48.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:48.103+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:48.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:48.104+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:49.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:48.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:48.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:48.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:48.710+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:49.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:49.080+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:49.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:49.080+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:49.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:49.080+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:49.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:49.080+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:49.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:49.278+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:49.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:49.278+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:49.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:49.278+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:49.334 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:49.279+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:50.046 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:49.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:50.046 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:49.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:50.046 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:49.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:50.046 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:49.734+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:50.313 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:50.043+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:50.313 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:50.043+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:50.313 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:50.043+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:50.313 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:50.044+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:50.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:50.311+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:50.311+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:50.311+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:50.312+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:51.062 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:50.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:51.063 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:50.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:51.063 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:50.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:51.063 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:50.769+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:51.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:51.061+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:51.061+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:51.061+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:51.062+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:51.298+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:51.298+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:51.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:51.298+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:51.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:51.299+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:52.041 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:51.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:52.041 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:51.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:52.042 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:51.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:52.042 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:51.770+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:52.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:12:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:12:52.100+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 18707 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:12:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:52.039+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:52.039+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:52.039+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:52.040+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:52.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:52.285+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:52.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:52.285+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:52.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:52.285+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:52.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:52.286+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:53.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:52.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:53.002 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:52.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:53.002 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:52.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:53.002 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:52.728+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:53.317 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:53.000+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:53.317 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:53.000+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:53.317 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:53.000+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:53.317 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:53.000+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:53.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:53.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:53.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:53.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:53.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:53.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:53.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:53.316+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:53.970 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:53.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:53.971 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:53.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:53.971 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:53.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:53.971 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:53.776+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:54.324 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:53.968+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:54.324 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:53.968+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:54.324 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:53.968+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:54.325 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:53.969+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:54.321+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:54.321+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:54.321+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:54.322+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:55.009 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:54.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:55.009 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:54.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:55.009 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:54.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:55.010 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:54.800+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:55.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:55.006+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:55.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:55.006+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:55.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:55.006+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:55.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:55.007+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:55.809 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:55.359+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:55.809 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:55.359+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:55.809 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:55.359+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:55.809 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:55.360+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:56.026+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:56.671 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:56.026+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:56.672 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:56.026+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:56.672 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:56.027+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:56.672 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:55.806+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:56.672 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:55.806+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:56.672 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:55.806+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:56.673 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:55.807+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:56.673 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:56.312+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:56.673 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:56.312+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:56.673 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:56.312+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:56.673 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:56.313+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:57.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:56.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:56.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:56.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:56.760+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:57.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:56.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:57.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:56.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:57.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:56.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:57.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:56.999+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:57.516 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:12:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:12:57.101+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 18765 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:12:57.517 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:57.301+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:57.517 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:57.301+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:57.517 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:57.301+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:57.517 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:57.301+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:58.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:57.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:57.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:57.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:57.809+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:58.029+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:58.029+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:58.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:58.029+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:58.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:58.030+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:58.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:58.296+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:58.296+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:58.296+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:58.297+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:59.312 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:58.834+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:59.313 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:58.834+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:59.313 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:58.834+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:12:59.313 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:58.835+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:12:59.313 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:58.983+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:59.313 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:58.983+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:59.313 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:58.983+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:12:59.314 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:12:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:12:58.983+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:12:59.314 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:59.309+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:59.314 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:59.309+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:59.314 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:59.309+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:12:59.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:12:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:12:59.311+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:00.291 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:00.016+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:00.291 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:00.016+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:00.292 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:00.016+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:00.292 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:00.017+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:00.292 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:59.879+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:00.292 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:59.879+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:00.293 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:59.879+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:00.293 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:12:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:12:59.880+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:00.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:00.290+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:00.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:00.290+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:00.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:00.290+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:00.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:00.291+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:01.153 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:00.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:01.153 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:00.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:01.154 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:00.872+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:01.154 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:00.873+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:01.154 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:01.019+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:01.154 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:01.019+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:01.154 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:01.019+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:01.154 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:01.020+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:01.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:01.293+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:01.293+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:01.293+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:01.294+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:02.105 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:01.979+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:02.106 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:01.979+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:02.106 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:01.979+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:02.106 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:01.980+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:02.107 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:01.844+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:02.107 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:01.844+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:02.107 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:01.844+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:02.107 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:01.845+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:02.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:13:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:13:02.104+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 18824 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:13:02.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:02.304+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:02.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:02.304+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:02.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:02.304+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:02.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:02.305+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:03.316 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:02.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:03.317 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:02.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:03.317 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:02.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:03.317 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:02.843+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:03.317 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:03.016+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:03.317 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:03.016+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:03.317 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:03.016+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:03.318 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:03.016+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:03.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:03.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:03.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:03.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:03.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:03.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:03.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:03.315+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:04.013 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:03.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:04.013 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:03.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:04.013 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:03.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:04.013 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:03.816+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:04.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:04.011+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:04.011+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:04.011+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:04.012+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:04.793 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:04.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:04.794 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:04.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:04.794 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:04.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:04.794 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:04.362+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:05.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:04.792+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:04.792+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:04.792+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:04.793+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:05.039+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:05.039+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:05.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:05.039+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:05.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:05.039+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:05.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:05.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:05.815 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:05.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:05.815 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:05.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:05.815 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:05.400+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:06.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:06.023+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:06.023+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:06.023+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:06.024+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:05.812+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:05.812+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:06.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:05.812+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:06.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:05.813+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:06.609 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:06.395+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:06.609 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:06.395+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:06.609 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:06.395+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:06.609 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:06.396+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:06.863 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:06.795+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:06.863 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:06.795+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:06.863 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:06.795+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:06.864 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:06.796+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:07.115 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:13:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:13:07.106+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 18883 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:13:07.116 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:07.055+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:07.116 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:07.055+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:07.116 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:07.055+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:07.116 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:07.056+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:07.819 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:07.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:07.819 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:07.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:07.819 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:07.374+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:07.820 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:07.375+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:08.078 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:07.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:08.078 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:07.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:08.079 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:07.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:08.079 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:07.818+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:08.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:08.077+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:08.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:08.077+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:08.077+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:08.077+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:08.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:08.390+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:08.390+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:08.390+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:08.390+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:09.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:09.064+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:09.064+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:09.064+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:09.064+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:08.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:08.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:08.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:09.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:08.863+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:09.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:09.375+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:09.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:09.375+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:09.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:09.375+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:09.683 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:09.376+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:10.022 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:09.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:10.022 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:09.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:10.022 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:09.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:10.022 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:09.875+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:10.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:10.020+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:10.020+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:10.020+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:10.021+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:10.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:10.421+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:10.421+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:10.421+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:10.422+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:11.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:10.878+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:10.878+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:10.878+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:10.879+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:11.029+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:11.029+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:11.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:11.029+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:11.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:11.029+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:11.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:11.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:11.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:11.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:11.417+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:12.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:13:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:13:12.108+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 18942 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:13:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:11.859+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:11.859+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:11.859+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:12.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:11.859+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:12.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:12.015+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:12.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:12.015+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:12.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:12.015+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:12.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:12.016+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:12.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:12.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:12.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:12.380+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:12.381+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:13.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:12.878+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:12.878+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:12.878+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:12.878+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:13.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:13.000+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:13.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:13.000+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:13.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:13.000+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:13.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:13.001+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:13.635 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:13.376+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:13.635 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:13.376+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:13.635 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:13.376+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:13.635 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:13.376+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:14.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:13.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:14.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:13.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:14.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:13.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:14.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:13.883+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:14.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:14.032+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:14.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:14.032+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:14.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:14.032+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:14.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:14.032+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:14.798 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:14.349+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:14.798 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:14.349+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:14.798 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:14.349+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:14.799 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:14.350+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:15.079 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:14.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:15.079 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:14.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:15.079 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:14.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:15.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:14.836+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:15.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:15.077+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:15.077+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:15.077+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:15.078+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:15.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:15.357+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:15.357+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:15.357+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:15.357+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:16.069+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:16.069+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:16.069+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:16.070+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:15.851+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:15.851+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:16.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:15.851+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:16.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:15.852+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:16.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:16.349+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:16.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:16.349+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:16.349+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:16.350+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:17.111 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:17.076+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:17.111 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:17.076+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:17.112 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:17.076+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:17.112 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:17.077+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:17.112 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:16.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:17.112 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:16.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:17.113 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:16.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:17.113 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:16.840+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:17.379 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:13:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:13:17.110+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19000 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:13:17.379 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:17.318+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:17.379 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:17.318+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:17.379 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:17.318+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:17.380 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:17.319+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:18.283 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:17.861+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:18.283 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:17.861+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:18.283 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:17.861+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:18.283 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:17.861+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:18.284 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:18.082+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:18.284 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:18.083+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:18.284 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:18.083+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:18.284 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:18.083+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:18.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:18.281+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:18.281+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:18.281+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:18.282+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:19.247 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:18.889+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:19.247 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:18.889+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:19.247 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:18.889+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:19.247 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:18.890+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:19.247 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:19.040+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:19.248 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:19.040+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:19.248 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:19.040+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:19.248 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:19.041+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:19.245+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:19.245+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:19.245+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:19.246+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:20.254 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:19.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:20.254 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:19.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:20.254 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:19.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:20.254 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:19.941+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:20.255 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:20.021+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:20.255 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:20.021+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:20.255 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:20.021+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:20.255 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:20.022+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:20.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:20.252+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:20.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:20.252+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:20.252+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:20.253+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:21.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:20.912+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:21.298 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:20.912+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:21.298 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:20.912+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:21.298 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:20.912+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:21.298 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:21.064+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:21.298 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:21.064+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:21.298 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:21.064+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:21.299 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:21.064+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:21.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:21.296+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:21.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:21.296+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:21.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:21.296+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:21.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:21.297+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:22.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:21.922+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:21.922+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:21.922+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:21.923+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:22.062+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:22.062+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:22.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:22.062+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:22.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:22.063+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:22.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:13:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:13:22.111+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19059 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:13:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:22.281+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:22.281+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:22.281+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:22.282+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:23.323 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:23.069+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:23.324 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:23.069+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:23.324 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:23.069+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:23.324 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:23.069+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:23.324 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:22.956+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:23.324 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:22.956+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:23.324 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:22.956+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:23.324 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:22.957+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:23.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:23.322+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:23.322+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:23.322+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:23.323+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:24.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:24.066+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:24.066+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:24.066+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:24.066+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:23.968+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:23.968+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:23.968+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:24.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:23.969+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:24.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:24.338+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:24.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:24.338+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:24.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:24.338+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:24.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:24.339+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:25.308 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:24.925+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:25.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:24.925+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:25.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:24.925+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:25.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:24.925+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:25.309 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:25.031+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:25.309 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:25.031+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:25.309 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:25.031+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:25.310 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:25.032+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:25.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:25.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:25.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:25.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:25.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:25.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:25.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:25.307+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:26.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:25.949+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:26.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:25.949+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:26.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:25.949+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:26.076 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:25.950+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:26.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:26.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:26.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:26.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:26.280+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:26.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:26.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:26.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:26.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:26.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:26.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:26.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:26.074+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:27.325 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:13:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:13:27.113+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19118 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:13:27.326 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:26.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:27.326 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:26.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:27.326 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:26.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:27.326 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:26.921+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:27.326 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:27.044+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:27.326 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:27.044+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:27.327 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:27.044+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:27.327 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:27.045+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:27.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:27.324+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:27.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:27.324+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:27.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:27.324+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:27.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:27.324+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:28.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:28.050+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:28.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:28.050+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:28.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:28.050+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:28.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:28.051+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:27.951+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:27.951+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:27.951+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:28.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:27.952+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:28.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:28.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:28.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:28.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:28.352+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:29.322 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:28.992+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:29.322 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:28.992+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:29.322 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:28.992+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:29.323 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:28.992+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:29.323 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:29.098+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:29.323 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:29.098+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:29.323 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:29.098+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:29.323 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:29.099+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:29.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:29.320+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:29.320+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:29.320+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:29.321+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:30.309 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:30.115+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:30.309 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:30.115+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:30.309 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:30.115+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:30.310 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:30.116+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:30.310 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:29.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:30.310 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:29.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:30.310 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:29.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:30.311 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:29.954+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:30.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:30.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:30.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:30.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:30.308+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:31.301 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:30.912+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:31.302 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:30.912+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:31.302 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:30.912+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:31.302 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:30.913+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:31.302 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:31.069+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:31.302 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:31.069+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:31.303 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:31.069+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:31.303 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:31.070+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:31.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:31.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:31.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:31.299+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:31.300+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:32.323 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:13:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:13:32.114+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19177 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:13:32.324 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:32.107+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:32.324 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:32.107+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:32.324 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:32.107+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:32.324 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:32.108+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:32.325 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:31.927+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:32.325 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:31.927+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:32.325 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:31.927+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:32.325 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:31.927+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:32.320+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:32.320+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:32.320+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:32.321+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:32.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:32.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:32.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:32.958+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:33.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:33.148+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:33.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:33.148+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:33.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:33.148+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:33.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:33.149+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:33.637 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:33.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:33.637 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:33.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:33.637 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:33.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:33.637 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:33.352+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:34.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:33.929+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:33.929+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:33.929+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:33.929+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:34.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:34.123+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:34.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:34.123+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:34.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:34.123+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:34.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:34.124+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:34.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:34.339+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:34.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:34.339+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:34.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:34.339+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:34.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:34.340+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:35.156 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:34.900+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:35.156 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:34.900+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:35.156 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:34.900+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:35.156 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:34.901+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:35.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:35.154+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:35.154+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:35.154+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:35.154+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:35.384+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:35.384+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:35.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:35.384+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:35.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:35.385+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:36.169 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:35.857+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:36.169 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:35.858+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:36.169 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:35.858+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:36.169 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:35.858+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:36.429 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:36.166+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:36.430 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:36.166+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:36.430 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:36.166+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:36.430 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:36.167+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:36.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:36.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:36.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:36.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:36.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:36.428+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:37.119 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:36.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:37.119 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:36.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:37.120 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:36.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:37.120 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:36.830+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:37.463 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:13:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:13:37.116+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19234 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:13:37.464 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:37.182+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:37.464 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:37.182+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:37.464 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:37.182+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:37.464 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:37.183+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:37.806 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:37.462+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:37.806 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:37.462+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:37.806 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:37.462+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:37.806 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:37.462+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:37.804+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:37.804+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:37.804+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:37.805+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:38.448 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:38.164+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:38.448 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:38.164+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:38.449 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:38.164+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:38.449 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:38.165+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:38.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:38.447+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:38.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:38.447+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:38.780 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:38.447+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:38.780 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:38.447+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:39.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:38.778+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:38.778+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:38.778+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:38.778+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:39.496 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:39.128+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:39.496 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:39.128+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:39.496 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:39.128+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:39.497 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:39.129+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:39.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:39.494+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:39.494+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:39.494+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:39.495+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:39.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:39.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:39.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:39.743+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:40.540 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:40.150+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:40.541 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:40.150+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:40.541 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:40.150+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:40.541 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:40.151+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:40.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:40.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:40.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:40.702+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:40.539+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:40.539+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:40.539+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:40.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:40.540+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:41.579 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:41.154+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:41.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:41.154+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:41.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:41.154+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:41.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:41.154+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:41.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:41.712+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:41.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:41.712+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:41.712+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:41.713+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:41.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:41.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:41.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:41.578+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:42.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:13:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:13:42.118+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19293 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:13:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:42.142+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:42.142+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:42.142+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:42.143+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:43.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:42.712+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:42.712+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:42.712+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:42.712+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:42.628+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:42.628+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:42.628+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:43.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:42.628+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:43.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:43.148+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:43.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:43.148+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:43.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:43.148+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:43.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:43.149+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:44.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:43.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:43.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:43.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:43.684+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:44.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:43.626+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:44.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:43.626+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:44.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:43.626+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:44.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:43.627+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:44.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:44.197+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:44.197+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:44.197+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:44.198+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:45.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:44.597+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:45.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:44.597+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:45.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:44.597+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:45.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:44.597+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:44.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:44.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:45.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:44.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:45.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:44.703+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:45.386 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:45.248+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:45.387 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:45.248+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:45.387 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:45.248+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:45.387 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:45.248+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:45.667 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:45.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:45.667 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:45.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:45.667 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:45.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:45.668 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:45.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:45.668 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:45.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:45.668 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:45.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:45.668 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:45.561+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:46.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:45.666+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:46.529 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:46.263+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:46.530 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:46.263+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:46.530 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:46.263+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:46.530 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:46.263+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:46.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:46.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:46.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:46.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:46.528+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:46.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:46.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:46.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:46.684+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:46.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:46.684+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:47.538 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:13:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:13:47.121+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19352 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:13:47.539 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:47.254+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:47.539 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:47.254+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:47.540 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:47.254+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:47.540 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:47.255+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:47.642+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:47.642+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:47.642+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:47.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:47.643+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:47.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:47.536+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:47.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:47.536+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:47.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:47.536+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:47.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:47.536+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:48.522 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:48.243+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:48.523 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:48.243+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:48.523 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:48.243+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:48.523 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:48.243+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:48.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:48.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:48.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:48.670+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:48.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:48.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:48.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:48.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:48.522+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:49.551 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:49.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:49.551 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:49.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:49.551 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:49.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:49.551 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:49.272+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:49.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:49.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:49.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:49.714+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:49.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:49.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:49.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:49.550+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:50.558 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:50.237+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:50.558 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:50.237+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:50.558 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:50.237+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:50.558 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:50.238+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:50.731+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:50.731+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:50.731+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:50.731+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:50.556+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:50.556+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:50.556+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:50.557+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:51.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:51.191+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:51.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:51.191+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:51.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:51.191+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:51.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:51.192+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:51.695+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:51.695+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:51.695+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:51.696+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:51.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:52.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:51.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:52.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:51.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:52.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:51.582+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:52.550 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:13:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:13:52.123+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19411 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:13:52.551 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:52.216+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:52.551 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:52.216+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:52.551 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:52.216+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:52.551 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:52.217+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:52.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:52.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:52.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:52.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:52.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:52.550+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:52.659+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:52.659+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:52.659+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:52.660+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:53.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:53.219+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:53.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:53.219+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:53.570 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:53.219+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:53.570 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:53.220+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:53.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:53.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:53.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:53.660+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:53.590+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:54.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:53.590+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:54.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:53.590+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:54.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:53.591+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:54.575 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:54.234+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:54.575 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:54.234+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:54.575 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:54.234+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:54.575 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:54.235+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:54.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:54.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:54.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:54.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:54.574+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:54.630+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:54.631+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:54.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:54.631+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:54.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:54.631+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:55.561 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:55.220+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:55.561 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:55.220+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:55.562 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:55.220+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:55.562 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:55.221+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:55.587+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:55.587+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:55.587+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:55.588+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:55.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:55.614+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:55.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:55.614+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:55.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:55.614+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:55.615+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:56.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:56.263+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:56.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:56.263+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:56.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:56.263+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:56.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:56.263+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:56.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:56.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:56.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:56.548+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:56.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:56.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:56.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:56.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:56.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:56.590+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:57.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:57.273+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:57.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:57.273+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:57.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:57.273+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:57.570 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:57.274+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:57.570 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:13:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:13:57.125+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19466 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:13:57.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:57.596+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:57.596+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:57.596+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:57.596+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:57.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:57.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:57.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:57.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:57.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:57.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:57.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:57.568+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:58.282+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:58.282+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:58.282+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:58.282+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:58.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:58.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:58.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:58.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:58.579+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:58.590+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:58.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:58.590+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:58.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:58.590+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:58.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:58.591+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:59.568 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:59.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:59.568 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:59.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:59.568 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:59.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:13:59.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:13:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:13:59.297+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:13:59.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:59.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:59.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:59.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:13:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:13:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:13:59.614+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:13:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:59.566+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:59.566+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:59.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:59.566+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:13:59.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:13:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:13:59.567+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:00.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:00.257+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:00.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:00.257+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:00.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:00.257+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:00.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:00.257+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:00.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:00.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:00.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:00.627+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:00.595+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:00.595+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:01.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:00.595+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:01.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:00.596+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:01.572 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:01.280+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:01.572 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:01.280+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:01.573 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:01.280+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:01.573 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:01.281+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:01.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:01.590+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:01.590+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:01.590+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:01.590+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:01.570+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:01.570+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:01.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:01.570+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:01.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:01.571+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:02.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:14:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:14:02.127+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19525 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:14:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:02.240+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:02.240+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:02.240+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:02.241+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:03.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:02.586+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:02.586+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:02.586+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:02.587+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:02.604+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:02.604+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:03.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:02.604+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:03.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:02.605+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:03.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:03.244+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:03.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:03.244+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:03.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:03.244+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:03.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:03.245+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:04.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:03.582+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:03.582+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:03.582+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:03.582+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:03.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:03.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:04.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:03.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:04.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:03.648+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:04.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:04.294+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:04.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:04.294+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:04.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:04.294+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:04.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:04.294+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:05.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:04.686+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:04.686+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:04.686+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:04.686+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:05.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:04.590+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:05.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:04.590+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:05.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:04.590+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:05.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:04.590+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:05.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:05.563+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:05.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:05.563+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:05.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:05.563+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:05.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:05.564+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:05.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:05.315+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:05.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:05.315+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:05.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:05.315+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:05.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:05.316+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:05.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:05.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:05.665+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:05.666+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:06.623 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:06.515+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:06.624 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:06.515+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:06.624 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:06.515+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:06.624 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:06.515+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:06.624 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:06.340+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:06.624 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:06.340+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:06.624 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:06.340+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:06.625 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:06.341+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:07.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:06.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:06.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:06.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:06.622+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:07.542 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:07.343+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:07.543 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:07.343+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:07.543 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:07.343+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:07.543 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:07.344+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:07.543 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:14:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:14:07.129+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19585 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:14:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:07.652+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:07.652+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:07.652+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:07.653+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:07.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:07.541+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:07.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:07.541+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:07.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:07.541+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:07.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:07.541+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:08.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:08.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:08.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:08.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:08.530+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:08.302+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:08.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:08.302+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:08.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:08.302+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:08.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:08.303+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:09.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:08.612+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:08.612+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:08.612+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:08.613+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:09.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:09.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:09.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:09.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:09.571+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:09.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:09.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:09.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:09.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:09.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:09.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:09.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:09.545+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:09.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:09.337+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:09.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:09.337+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:09.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:09.337+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:09.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:09.338+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:10.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:10.366+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:10.366+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:10.366+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:10.367+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:10.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:10.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:10.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:10.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:35.607548+0000 (oldest deadline 2024-09-10T04:03:55.607548+0000) 2024-09-10T04:14:10.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:10.528+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:10.541+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:10.541+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:10.541+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:10.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:10.541+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:11.810 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:11.544+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:11.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:11.571+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:11.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:11.571+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:11.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:11.571+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:11.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:11.572+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:11.811 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:11.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:11.811 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:11.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:11.812 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:11.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:11.812 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:11.362+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:12.530 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:14:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:14:12.130+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19644 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:14:12.530 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:12.363+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:12.531 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:12.363+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:12.531 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:12.363+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:12.531 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:12.364+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:12.528+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:12.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:12.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:12.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:12.579+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:13.629 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:13.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:13.630 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:13.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:13.630 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:13.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:13.630 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:13.375+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:13.630 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:13.531+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:14.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:13.627+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:13.627+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:13.627+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:13.628+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:14.666 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:14.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:14.667 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:14.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:14.667 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:14.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.881750+0000 (oldest deadline 2024-09-10T04:04:05.881750+0000) 2024-09-10T04:14:14.667 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:14.404+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:14.667 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:14.498+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:15.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:14.665+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:15.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:14.665+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:15.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:14.665+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:15.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:14.666+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:15.702 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:15.498+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:15.702 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:15.426+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:16.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:15.700+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:15.700+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:15.700+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:15.701+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:16.752 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:16.453+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:16.753 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:16.493+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:17.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:16.751+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:16.751+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:16.751+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:16.752+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:17.419 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:14:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:14:17.133+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19702 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:14:17.709 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:17.461+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:17.709 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:17.417+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:17.960 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:17.707+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:17.960 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:17.707+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:17.960 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:17.707+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:17.961 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:17.708+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:18.703 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:18.445+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:18.704 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:18.444+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:19.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:18.702+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:18.702+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:18.702+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:03:45.995091+0000 (oldest deadline 2024-09-10T04:04:05.995091+0000) 2024-09-10T04:14:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:18.702+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:19.693 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:19.425+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:19.693 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:19.473+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:20.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:19.691+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:20.741 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:20.486+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:20.741 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:20.376+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:21.006 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:20.739+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:21.709 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:21.426+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:21.709 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:21.466+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:22.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:21.707+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:22.421 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:14:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:14:22.135+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19761 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:14:22.679 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:22.419+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:22.679 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:22.503+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:23.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:22.677+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:23.777 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:23.519+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:23.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:23.402+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:23.778 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:23.644+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:24.682 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:24.529+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:24.682 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:24.406+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:25.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:24.680+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:25.513+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:25.679+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:25.454+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:26.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:26.501+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:26.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:26.543+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:26.653+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:27.454 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:14:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:14:27.137+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19820 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:14:27.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:27.631+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:27.543+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:27.452+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:28.532+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:28.412+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:28.629+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:29.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:29.525+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:29.386+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:29.593+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:30.557 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:30.349+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:30.556+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:30.641+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:31.556 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:31.376+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:31.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:31.554+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:31.656+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:32.571 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:14:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:14:32.139+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19878 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:14:32.571 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:32.380+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:32.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:32.569+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:32.692+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:33.639 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:33.532+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:33.640 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:33.346+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:34.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:33.716+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:34.571 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:34.315+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:34.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:34.757+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:34.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:34.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:34.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:34.570+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:35.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:35.303+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:35.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:35.521+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:35.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:35.521+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:35.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:35.521+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:35.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:35.521+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:36.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:35.795+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:36.530 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:36.285+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:36.798 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:36.529+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:36.798 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:36.529+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:36.798 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:36.529+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:36.798 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:36.529+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:37.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:36.796+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:37.550 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:37.326+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:37.550 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:14:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:14:37.141+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19937 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:14:37.807 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:37.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:37.807 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:37.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:37.807 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:37.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:37.808 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:37.549+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:38.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:37.806+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:38.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:38.334+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:38.542+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:38.542+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:38.542+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:38.543+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:38.843+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:39.649 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:39.343+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:39.649 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:39.343+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:39.649 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:39.343+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:39.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:39.344+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:39.650 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:39.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:39.650 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:39.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:39.650 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:39.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:39.651 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:39.592+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:40.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:39.845+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:40.599 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:40.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:40.600 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:40.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:40.600 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:40.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:40.600 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:40.342+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:41.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:40.598+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:40.598+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:40.598+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:40.599+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:41.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:40.811+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:41.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:40.811+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:41.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:40.811+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:41.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:40.811+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:41.812 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:41.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:41.813 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:41.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:41.813 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:41.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:41.813 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:41.569+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:41.813 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:41.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:41.813 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:41.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:41.813 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:41.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:41.814 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:41.367+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:42.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:41.811+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:42.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:41.811+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:42.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:41.811+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:42.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:41.811+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:42.522 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:14:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:14:42.143+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 19995 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:14:42.522 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:42.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:42.523 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:42.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:42.523 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:42.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:42.523 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:42.341+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:42.803 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:42.520+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:42.804 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:42.520+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:42.804 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:42.520+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:42.804 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:42.521+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:43.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:42.801+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:42.801+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:42.801+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:42.802+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:43.573 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:43.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:43.573 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:43.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:43.573 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:43.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:43.573 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:43.313+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:43.571+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:43.571+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:43.571+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:43.572+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:44.316 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:43.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:44.316 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:43.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:44.317 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:43.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:44.317 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:43.846+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:44.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:44.314+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:44.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:44.314+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:44.314+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:44.315+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:45.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:44.590+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:44.590+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:44.590+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:44.591+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:45.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:44.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:45.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:44.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:45.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:44.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:45.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:44.827+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:45.548 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:45.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:45.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:45.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:45.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:45.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:45.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:45.313+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:45.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:45.546+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:45.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:45.546+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:45.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:45.546+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:45.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:45.547+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:46.285 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:45.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:46.286 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:45.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:46.286 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:45.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:46.286 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:45.876+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:46.546 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:46.283+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:46.546 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:46.283+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:46.546 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:46.283+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:46.547 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:46.283+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:46.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:46.543+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:46.543+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:46.543+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:46.544+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:47.147 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:46.887+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:47.147 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:46.887+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:47.147 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:46.887+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:47.147 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:46.887+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:47.552 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:47.270+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:47.552 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:47.270+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:47.552 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:47.270+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:47.552 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:47.270+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:47.553 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:14:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:14:47.144+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 20054 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:14:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:47.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:47.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:47.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:47.550+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:48.311 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:47.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:48.312 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:47.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:48.312 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:47.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:48.312 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:47.871+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:48.568 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:48.309+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:48.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:48.309+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:48.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:48.309+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:48.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:48.309+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:48.565+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:48.565+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:48.565+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:48.566+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:49.271 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:48.901+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:49.271 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:48.901+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:49.271 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:48.901+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:49.271 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:48.902+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:49.574 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:49.268+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:49.574 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:49.268+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:49.575 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:49.268+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:49.575 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:49.269+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:49.572+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:49.572+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:49.572+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:49.572+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:50.292 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:49.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:50.292 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:49.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:50.292 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:49.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:50.292 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:49.856+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:50.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:50.289+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:50.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:50.289+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:50.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:50.289+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:50.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:50.290+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:50.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:50.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:50.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:50.567+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:51.254 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:50.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:51.254 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:50.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:51.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:50.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:51.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:50.879+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:51.545 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:51.252+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:51.545 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:51.252+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:51.546 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:51.252+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:51.546 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:51.252+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:51.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:51.608+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:51.608+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:51.608+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:51.609+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:52.213 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:51.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:52.213 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:51.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:52.214 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:51.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:52.214 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:51.928+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:52.214 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:14:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:14:52.147+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 20113 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:14:52.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:52.211+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:52.211+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:52.211+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:52.212+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:52.881 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:52.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:52.881 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:52.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:52.882 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:52.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:52.882 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:52.605+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:53.206 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:52.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:53.207 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:52.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:53.207 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:52.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:53.207 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:52.880+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:53.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:53.205+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:53.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:53.205+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:53.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:53.205+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:53.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:53.206+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:53.901 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:53.624+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:53.901 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:53.624+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:53.901 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:53.624+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:53.902 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:53.625+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:54.241 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:53.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:54.241 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:53.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:54.242 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:53.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:54.242 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:53.900+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:54.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:54.240+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:54.240+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:54.240+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:54.240+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:54.907 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:54.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:54.908 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:54.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:54.908 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:54.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:54.908 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:54.665+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:55.242 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:54.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:55.243 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:54.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:55.243 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:54.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:55.243 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:54.907+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:55.521 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:55.241+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:55.521 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:55.241+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:55.522 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:55.241+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:55.522 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:55.241+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:56.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:55.678+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:55.678+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:55.678+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:55.679+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:55.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:56.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:55.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:56.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:55.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:56.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:55.891+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:56.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:56.255+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:56.255+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:56.255+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:56.256+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:57.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:56.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:56.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:56.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:56.638+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:56.873+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:57.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:56.873+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:57.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:56.873+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:57.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:56.874+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:57.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:14:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:14:57.149+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 20172 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:14:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:57.275+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:57.275+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:57.275+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:57.276+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:58.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:57.666+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:57.666+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:57.666+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:57.667+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:57.864+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:57.864+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:58.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:57.864+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:58.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:57.864+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:58.265+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:58.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:58.265+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:58.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:58.265+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:58.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:58.266+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:58.916 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:58.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:58.916 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:58.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:58.917 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:58.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:58.917 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:58.618+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:59.222 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:58.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:59.222 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:58.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:59.223 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:58.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:14:59.223 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:58.915+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:14:59.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:59.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:59.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:59.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:59.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:59.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:14:59.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:14:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:14:59.221+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:14:59.938 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:59.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:59.938 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:59.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:59.939 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:59.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:14:59.939 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:14:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:14:59.605+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:00.233 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:59.936+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:00.234 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:59.936+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:00.234 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:59.936+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:00.234 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:14:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:14:59.937+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:00.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:00.230+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:00.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:00.230+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:00.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:00.230+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:00.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:00.231+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:00.922 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:00.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:00.922 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:00.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:00.923 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:00.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:00.923 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:00.596+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:01.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:00.921+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:01.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:00.921+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:01.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:00.921+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:01.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:00.922+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:01.571 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:01.258+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:01.571 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:01.258+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:01.571 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:01.258+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:01.571 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:01.259+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:01.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:01.569+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:01.569+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:01.569+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:01.570+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:02.153 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:01.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:02.153 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:01.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:02.154 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:01.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:02.154 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:01.894+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:02.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:15:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:15:02.151+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 20231 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:15:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:02.231+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:02.231+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:02.231+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:02.231+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:02.619+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:03.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:02.619+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:03.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:02.619+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:03.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:02.620+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:03.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:02.856+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:03.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:02.856+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:03.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:02.856+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:03.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:02.857+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:03.577 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:03.234+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:03.578 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:03.234+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:03.578 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:03.234+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:03.578 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:03.234+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:03.576+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:03.576+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:03.576+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:03.577+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:03.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:03.809+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:03.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:03.809+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:03.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:03.809+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:03.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:03.809+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:04.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:04.202+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:04.202+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:04.202+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:04.202+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:04.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:04.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:04.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:05.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:04.592+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:05.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:04.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:05.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:04.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:05.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:04.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:05.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:04.766+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:05.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:05.177+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:05.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:05.177+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:05.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:05.177+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:05.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:05.178+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:06.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:05.620+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:05.620+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:05.620+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:05.620+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:05.810+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:05.810+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:06.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:05.810+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:06.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:05.811+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:06.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:06.178+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:06.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:06.178+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:06.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:06.178+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:06.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:06.179+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:07.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:06.632+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:06.632+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:06.632+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:06.633+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:07.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:06.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:07.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:06.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:07.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:06.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:07.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:06.766+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:07.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:15:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:15:07.153+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 20289 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:15:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:07.207+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:07.207+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:07.207+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:07.207+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:08.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:07.718+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:07.718+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:07.718+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:07.718+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:07.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:07.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:08.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:07.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:08.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:07.660+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:08.177+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:08.177+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:08.177+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:08.177+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:09.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:08.763+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:09.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:08.763+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:09.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:08.763+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:09.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:08.764+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:08.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:08.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:08.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:09.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:08.688+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:09.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:09.159+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:09.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:09.159+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:09.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:09.159+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:09.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:09.160+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:09.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:09.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:09.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:09.683+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:10.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:09.803+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:10.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:09.803+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:10.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:09.803+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:10.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:09.804+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:10.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:10.152+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:10.152+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:10.152+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:10.153+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:11.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:10.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:10.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:10.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:10.668+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:11.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:10.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:11.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:10.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:11.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:10.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:11.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:10.766+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:11.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:11.120+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:11.120+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:11.120+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:11.121+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:12.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:11.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:12.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:11.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:12.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:11.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:12.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:11.710+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:11.790+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:11.790+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:12.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:11.790+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:12.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:11.790+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:12.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:15:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:15:12.155+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 20347 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:15:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:12.138+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:12.138+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:12.138+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:12.139+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:13.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:12.751+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:13.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:12.751+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:13.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:12.751+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:13.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:12.751+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:12.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:12.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:12.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:13.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:12.758+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:13.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:13.176+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:13.176+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:13.176+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:13.177+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:14.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:13.731+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:14.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:13.731+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:13.731+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:13.732+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:13.795+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:13.795+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:13.795+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:13.796+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:14.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:14.218+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:14.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:14.218+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:14.218+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:14.219+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:15.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:14.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:14.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:14.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:14.719+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:15.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:14.775+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:15.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:14.775+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:15.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:14.775+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:15.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:14.775+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:15.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:15.191+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:15.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:15.191+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:15.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:15.191+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:15.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:15.192+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:16.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:15.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:15.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:15.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:15.737+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:15.769+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:15.769+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:15.769+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:16.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:15.770+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:16.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:16.179+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:16.179+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:16.179+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:16.180+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:17.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:16.814+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:16.814+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:16.814+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:16.815+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:16.780+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:16.780+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:16.780+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:17.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:16.781+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:17.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:15:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:15:17.157+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 20405 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:15:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:17.225+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:17.225+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:17.225+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:17.226+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:18.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:17.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:18.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:17.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:18.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:17.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:18.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:17.806+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:18.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:17.800+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:18.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:17.800+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:18.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:17.800+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:18.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:17.800+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:18.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:18.236+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:18.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:18.236+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:18.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:18.236+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:18.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:18.237+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:18.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:18.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:18.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:18.805+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:18.785+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:18.785+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:19.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:18.785+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:19.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:18.786+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:19.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:19.235+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:19.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:19.235+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:19.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:19.235+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:19.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:19.236+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:19.827+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:19.827+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:19.827+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:19.827+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:19.737+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:19.737+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:19.737+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:19.738+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:20.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:20.222+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:20.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:20.222+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:20.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:20.222+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:20.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:20.223+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:20.771+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:20.771+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:20.771+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:21.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:20.771+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:21.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:20.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:21.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:20.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:21.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:20.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:21.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:20.817+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:21.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:21.264+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:21.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:21.264+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:21.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:21.264+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:21.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:21.264+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:22.066 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:21.810+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:22.066 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:21.810+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:22.066 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:21.810+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:22.066 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:21.811+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:22.067 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:21.734+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:22.067 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:21.734+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:22.067 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:21.734+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:22.067 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:21.735+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:22.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:15:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:15:22.159+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 20464 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:15:22.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:22.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:22.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:22.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:22.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:22.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:22.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:22.239+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:22.834+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:22.834+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:22.834+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:22.834+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:23.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:22.733+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:23.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:22.733+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:23.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:22.733+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:23.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:22.734+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:23.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:23.277+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:23.277+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:23.277+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:23.278+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:24.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:23.824+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:23.824+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:23.824+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:23.826+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:24.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:23.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:24.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:23.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:24.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:23.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:24.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:23.766+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:24.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:24.294+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:24.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:24.294+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:24.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:24.294+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:24.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:24.294+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:25.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:24.729+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:25.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:24.729+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:25.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:24.729+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:25.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:24.730+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:24.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:24.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:25.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:24.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:25.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:24.806+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:25.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:25.321+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:25.321+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:25.321+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:25.322+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:26.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:25.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:25.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:25.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:25.797+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:26.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:25.749+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:26.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:25.750+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:26.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:25.750+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:26.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:25.750+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:26.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:26.276+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:26.276+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:26.276+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:26.276+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:27.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:26.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:26.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:26.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:26.804+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:26.733+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:26.733+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:27.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:26.733+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:27.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:26.734+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:27.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:15:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:15:27.160+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 20523 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:15:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:27.297+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:27.297+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:27.297+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:27.298+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:28.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:27.723+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:27.723+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:27.723+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:27.724+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:27.801+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:27.801+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:27.801+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:28.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:27.801+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:28.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:28.253+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:28.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:28.253+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:28.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:28.253+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:28.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:28.254+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:28.783+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:28.783+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:29.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:28.783+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:29.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:28.783+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:29.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:28.711+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:29.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:28.711+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:29.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:28.711+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:29.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:28.712+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:29.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:29.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:29.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:29.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:29.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:29.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:29.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:29.271+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:29.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:29.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:29.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:29.830+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:29.735+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:29.735+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:29.735+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:30.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:29.736+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:30.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:30.224+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:30.224+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:30.224+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:30.224+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:30.879+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:30.879+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:30.879+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:30.880+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:30.752+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:30.752+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:30.752+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:31.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:30.752+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:31.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:31.252+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:31.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:31.252+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:31.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:31.252+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:31.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:31.253+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:32.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:31.907+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:31.907+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:31.907+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:31.907+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:31.723+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:31.723+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:31.723+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:31.724+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:32.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:15:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:15:32.163+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 20582 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:15:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:32.211+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:32.211+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:32.211+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:32.212+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:32.860+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:32.860+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:32.860+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:32.860+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:33.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:32.684+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:33.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:32.684+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:33.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:32.684+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:33.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:32.685+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:33.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:33.219+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:33.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:33.219+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:33.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:33.219+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:33.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:33.220+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:34.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:33.853+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:33.853+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:33.853+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:33.854+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:33.656+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:33.656+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:33.656+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:34.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:33.657+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:34.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:34.199+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:34.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:34.199+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:34.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:34.199+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:34.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:34.200+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:35.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:34.836+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:34.836+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:34.836+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:34.837+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:35.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:34.694+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:35.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:34.694+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:35.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:34.694+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:35.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:34.695+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:35.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:35.193+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:35.193+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:35.193+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:35.194+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:36.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:35.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:35.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:35.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:35.823+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:36.084 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:35.722+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:36.084 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:35.722+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:36.084 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:35.722+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:36.084 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:35.722+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:36.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:36.237+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:36.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:36.237+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:36.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:36.237+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:36.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:36.237+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:37.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:36.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:36.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:36.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:36.821+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:36.726+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:37.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:36.726+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:37.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:36.726+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:37.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:36.727+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:37.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:15:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:15:37.165+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 20639 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:15:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:37.285+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:37.285+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:37.285+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:37.286+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:37.808+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:37.808+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:37.808+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:37.809+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:38.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:37.760+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:38.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:37.760+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:38.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:37.760+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:38.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:37.761+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:38.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:38.265+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:38.265+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:38.265+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:38.266+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:39.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:38.790+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:38.790+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:38.790+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:38.791+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:38.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:39.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:38.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:39.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:38.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:39.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:38.838+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:39.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:39.251+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:39.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:39.251+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:39.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:39.251+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:39.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:39.252+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:40.265 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:39.865+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:40.265 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:39.865+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:40.265 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:39.865+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:40.266 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:39.866+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:40.266 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:39.836+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:40.266 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:39.836+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:40.266 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:39.836+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:40.266 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:39.837+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:40.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:40.263+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:40.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:40.263+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:40.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:40.263+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:40.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:40.264+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:41.218 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:40.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:41.218 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:40.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:41.218 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:40.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:41.219 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:40.909+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:41.219 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:40.835+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:41.219 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:40.835+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:41.219 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:40.835+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:41.219 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:40.836+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:41.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:41.216+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:41.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:41.216+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:41.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:41.216+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:41.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:41.217+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:42.169 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:41.939+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:42.170 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:41.939+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:42.170 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:41.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:42.170 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:41.940+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:42.170 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:41.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:42.170 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:41.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:42.170 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:41.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:42.171 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:41.857+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:42.433 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:15:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:15:42.168+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 20698 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:15:42.434 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:42.252+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:42.434 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:42.252+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:42.434 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:42.252+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:42.434 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:42.253+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:43.221 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:42.941+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:43.221 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:42.941+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:43.221 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:42.941+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:43.221 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:42.942+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:43.222 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:42.850+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:43.222 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:42.850+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:43.222 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:42.850+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:43.222 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:42.850+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:43.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:43.219+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:43.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:43.219+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:43.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:43.219+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:43.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:43.220+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:43.900 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:43.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:43.900 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:43.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:43.900 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:43.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:43.901 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:43.856+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:44.229 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:43.898+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:44.229 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:43.898+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:44.229 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:43.898+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:44.230 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:43.899+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:44.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:44.228+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:44.228+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:44.228+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:44.228+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:45.231 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:44.926+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:45.231 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:44.926+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:45.231 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:44.926+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:45.232 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:44.927+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:45.232 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:44.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:45.232 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:44.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:45.232 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:44.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:45.232 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:44.898+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:45.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:45.229+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:45.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:45.229+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:45.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:45.229+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:45.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:45.230+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:46.277 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:45.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:46.277 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:45.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:46.277 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:45.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:46.277 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:45.949+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:46.278 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:45.969+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:46.278 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:45.969+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:46.278 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:45.969+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:46.278 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:45.969+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:46.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:46.275+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:46.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:46.275+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:46.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:46.275+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:46.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:46.276+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:47.308 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:15:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:15:47.170+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 20757 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:15:47.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:46.941+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:47.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:46.941+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:47.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:46.941+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:47.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:46.941+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:47.309 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:46.981+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:47.310 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:46.981+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:47.310 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:46.981+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:47.310 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:46.981+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:47.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:47.307+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:47.307+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:47.307+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:47.308+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:48.263 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:47.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:48.263 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:47.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:48.263 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:47.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:48.263 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:47.976+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:48.264 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:47.956+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:48.264 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:47.956+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:48.264 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:47.956+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:48.264 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:47.957+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:48.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:48.261+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:48.261+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:48.261+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:48.262+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:49.222 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:48.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:49.223 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:48.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:49.223 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:48.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:49.223 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:48.966+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:49.224 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:48.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:49.224 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:48.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:49.224 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:48.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:49.224 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:48.965+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:49.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:49.220+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:49.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:49.220+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:49.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:49.220+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:49.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:49.221+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:50.248 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:49.974+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:50.248 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:49.974+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:50.248 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:49.974+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:50.249 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:49.974+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:50.249 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:50.012+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:50.249 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:50.012+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:50.250 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:50.012+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:50.250 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:50.013+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:50.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:50.246+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:50.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:50.246+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:50.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:50.246+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:50.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:50.247+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:51.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:50.984+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:51.298 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:50.984+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:51.298 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:50.984+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:51.298 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:50.985+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:51.298 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:51.047+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:51.298 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:51.047+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:51.299 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:51.047+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:51.299 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:51.048+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:51.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:51.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:51.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:51.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:51.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:51.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:51.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:51.297+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:52.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:52.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:52.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:52.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:52.021+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:52.000+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:52.000+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:52.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:52.000+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:52.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:52.000+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:52.332 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:15:52.172+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 20816 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:15:52.611 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:52.332+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:52.611 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:52.332+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:52.611 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:52.332+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:52.611 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:52.332+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:53.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:52.984+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:52.984+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:52.984+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:52.984+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:52.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:53.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:52.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:53.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:52.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:53.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:52.992+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:53.645 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:53.338+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:53.646 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:53.338+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:53.646 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:53.338+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:53.646 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:53.338+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:54.293 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:53.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:54.293 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:53.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:54.294 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:53.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:54.294 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:53.958+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:54.294 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:53.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:54.294 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:53.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:54.294 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:53.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:54.295 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:53.961+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:54.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:54.291+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:54.291+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:54.291+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:54.292+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:55.295 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:54.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:55.295 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:54.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:55.295 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:54.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:55.296 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:54.993+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:55.296 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:54.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:55.296 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:54.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:55.296 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:54.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:55.297 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:54.940+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:55.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:55.293+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:55.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:55.293+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:55.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:55.293+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:55.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:55.293+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:56.256 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:55.949+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:56.257 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:55.949+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:56.257 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:55.949+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:56.257 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:55.950+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:56.257 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:55.982+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:56.257 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:55.982+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:56.258 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:55.982+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:56.258 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:55.983+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:56.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:56.261+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:56.261+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:56.261+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:56.262+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:57.177 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:56.905+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:57.177 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:56.905+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:57.177 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:56.905+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:57.177 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:56.905+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:57.178 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:56.982+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:57.178 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:56.982+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:57.178 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:56.982+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:57.178 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:56.982+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:57.473 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:15:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:15:57.174+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 20874 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:15:57.474 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:57.213+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:57.474 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:57.213+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:57.474 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:57.213+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:57.474 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:57.214+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:57.988 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:57.919+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:57.989 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:57.919+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:57.989 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:57.919+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:57.989 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:57.919+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:58.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:58.219+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:58.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:58.219+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:58.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:58.219+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:58.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:58.219+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:15:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:57.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:58.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:57.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:58.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:57.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:58.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:57.986+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:59.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:58.946+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:59.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:58.946+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:59.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:58.946+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:15:59.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:58.947+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:59.271 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:58.938+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:59.272 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:58.938+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:59.272 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:58.938+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:15:59.272 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:58.938+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:15:59.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:59.269+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:59.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:59.269+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:59.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:59.269+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:15:59.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:15:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:15:59.270+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:00.282 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:59.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:00.283 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:59.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:00.283 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:59.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:00.283 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:15:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:15:59.989+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:00.283 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:59.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:00.283 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:59.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:00.283 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:59.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:00.284 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:15:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:15:59.951+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:00.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:00.280+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:00.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:00.280+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:00.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:00.280+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:00.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:00.280+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:01.312 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:00.968+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:01.312 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:00.968+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:01.312 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:00.968+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:01.313 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:00.968+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:01.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:00.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:01.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:00.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:01.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:00.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:01.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:00.955+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:01.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:01.309+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:01.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:01.309+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:01.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:01.309+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:01.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:01.310+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:02.277 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:16:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:16:02.175+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 20933 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:16:02.278 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:01.936+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:02.278 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:01.936+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:02.278 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:01.936+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:02.278 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:01.937+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:02.278 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:01.959+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:02.279 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:01.959+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:02.279 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:01.959+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:02.279 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:01.960+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:02.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:02.275+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:02.275+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:02.275+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:02.275+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:02.970+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:02.970+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:02.970+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:02.970+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:02.910+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:02.910+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:03.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:02.910+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:03.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:02.911+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:03.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:03.281+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:03.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:03.281+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:03.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:03.281+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:03.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:03.282+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:04.237 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:03.955+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:04.238 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:03.955+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:04.238 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:03.955+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:04.238 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:03.956+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:04.238 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:03.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:04.239 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:03.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:04.239 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:03.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:04.239 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:03.940+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:04.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:04.234+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:04.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:04.234+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:04.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:04.234+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:04.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:04.235+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:05.229 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:05.226+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:05.229 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:05.226+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:05.230 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:05.226+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:05.230 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:04.986+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:05.230 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:04.986+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:05.230 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:04.986+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:05.231 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:04.987+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:05.231 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:04.976+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:05.231 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:04.976+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:05.231 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:04.976+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:05.231 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:04.977+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:05.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:05.226+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:06.265 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:05.986+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:06.266 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:05.986+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:06.266 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:05.986+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:06.266 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:05.987+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:06.266 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:05.933+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:06.267 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:05.933+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:06.267 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:05.933+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:06.267 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:05.933+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:06.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:06.264+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:06.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:06.264+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:06.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:06.264+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:06.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:06.265+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:07.180 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:07.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:07.180 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:07.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:07.180 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:07.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:07.181 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:07.006+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:07.181 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:06.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:07.181 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:06.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:07.181 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:06.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:07.181 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:06.929+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:07.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:16:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:16:07.178+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 20992 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:16:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:07.233+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:07.233+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:07.233+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:07.234+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:08.276 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:08.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:08.276 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:08.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:08.277 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:08.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:08.277 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:08.017+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:08.277 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:07.965+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:08.277 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:07.965+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:08.277 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:07.965+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:08.277 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:07.966+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:08.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:08.274+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:08.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:08.274+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:08.274+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:08.275+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:09.286 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:08.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:09.286 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:08.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:09.286 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:08.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:09.286 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:08.964+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:09.286 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:09.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:09.287 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:09.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:09.287 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:09.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:09.287 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:09.007+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:09.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:09.284+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:09.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:09.284+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:09.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:09.284+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:09.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:09.285+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:10.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:09.997+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:09.997+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:09.997+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:09.997+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:09.985+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:09.985+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:10.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:09.985+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:10.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:09.986+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:10.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:10.329+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:10.329+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:10.329+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:10.329+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:10.986+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:11.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:10.986+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:11.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:10.986+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:11.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:10.987+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:10.980+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:10.980+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:10.980+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:10.981+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:11.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:11.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:11.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:11.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:11.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:11.331+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:12.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:11.957+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:12.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:11.957+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:12.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:11.957+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:12.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:11.957+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:12.334 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:11.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:12.334 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:11.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:12.334 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:11.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:12.334 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:11.961+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:12.334 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:16:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:16:12.180+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 21048 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:16:12.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:12.337+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:12.337+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:12.337+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:12.337+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:13.322 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:12.924+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:13.322 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:12.924+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:13.322 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:12.924+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:13.322 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:12.925+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:13.323 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:12.961+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:13.323 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:12.961+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:13.323 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:12.961+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:13.323 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:12.962+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:13.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:13.320+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:13.320+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:13.320+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:13.321+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:13.943 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:13.876+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:13.943 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:13.876+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:13.943 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:13.876+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:13.944 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:13.876+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:14.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:13.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:14.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:13.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:14.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:13.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:14.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:13.943+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:14.689 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:14.355+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:14.689 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:14.355+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:14.689 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:14.355+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:14.689 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:14.356+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:14.954 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:14.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:14.955 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:14.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:14.955 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:14.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:14.955 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:14.837+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:15.320 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:14.953+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:15.320 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:14.953+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:15.321 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:14.953+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:15.321 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:14.954+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:15.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:15.318+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:15.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:15.319+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:15.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:15.319+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:15.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:15.319+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:16.317 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:15.845+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:16.318 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:15.845+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:16.318 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:15.845+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:16.318 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:15.846+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:16.318 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:15.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:16.318 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:15.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:16.319 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:15.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:16.319 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:16.000+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:16.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:16.316+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:16.316+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:16.316+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:16.317+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:17.184 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:16.890+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:17.184 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:16.890+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:17.184 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:16.890+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:17.185 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:16.891+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:17.185 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:16.977+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:17.185 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:16.977+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:17.185 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:16.977+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:17.185 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:16.978+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:17.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:16:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:16:17.182+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 21099 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:16:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:17.337+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:17.337+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:17.337+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:17.337+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:18.014 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:17.859+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:18.014 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:17.859+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:18.014 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:17.859+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:18.014 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:17.860+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:18.015 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:17.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:18.015 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:17.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:18.015 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:17.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:18.015 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:17.986+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:18.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:18.371+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:18.371+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:18.371+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:18.372+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:19.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:18.968+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:19.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:18.968+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:19.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:18.968+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:19.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:18.969+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:18.850+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:18.850+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:19.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:18.850+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:19.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:18.851+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:19.805 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:19.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:19.805 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:19.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:19.805 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:19.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:19.805 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:19.408+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:19.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:19.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:19.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:19.804+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:19.996+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:19.996+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:20.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:19.996+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:20.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:19.996+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:20.829 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:20.455+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:20.829 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:20.455+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:20.455+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:20.456+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:21.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:20.828+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:20.828+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:20.828+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:20.828+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:21.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:21.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:21.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:21.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:21.043+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:21.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:21.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:21.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:21.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:21.419+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:22.186 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:21.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:22.187 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:21.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:22.187 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:21.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:22.187 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:21.838+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:22.187 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:22.078+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:22.187 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:22.078+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:22.187 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:22.078+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:22.187 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:22.079+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:22.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:16:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:16:22.185+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 21148 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:16:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:22.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:22.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:22.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:22.379+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:23.102 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:23.071+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:23.102 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:23.071+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:23.102 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:23.071+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:23.102 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:23.072+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:23.103 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:22.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:23.103 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:22.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:23.103 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:22.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:23.103 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:22.840+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:23.648 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:23.372+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:23.648 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:23.372+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:23.648 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:23.372+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:23.649 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:23.372+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:23.904 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:23.834+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:23.905 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:23.834+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:23.905 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:23.834+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:23.905 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:23.835+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:24.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:24.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:24.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:24.103+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:24.597 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:24.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:24.598 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:24.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:24.598 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:24.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:24.598 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:24.342+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:25.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:24.800+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:25.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:24.800+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:24.800+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:24.800+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:25.381 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:25.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:25.381 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:25.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:25.381 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:25.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:25.382 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:25.104+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:25.752 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:25.380+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:25.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:25.380+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:25.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:25.380+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:25.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:25.380+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:26.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:25.751+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:25.751+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:25.751+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:25.752+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:26.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:26.122+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:26.122+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:26.122+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:26.123+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:26.336+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:26.336+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:26.336+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:26.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:26.337+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:27.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:26.781+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:26.781+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:26.781+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:26.782+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:27.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:16:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:16:27.187+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 21197 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:16:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:27.304+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:27.304+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:27.304+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:27.305+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:27.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:27.106+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:27.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:27.106+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:27.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:27.106+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:27.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:27.107+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:27.815+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:27.815+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:27.815+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:27.816+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:28.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:28.297+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:28.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:28.297+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:28.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:28.297+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:28.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:28.298+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:28.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:28.098+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:28.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:28.098+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:28.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:28.098+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:28.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:28.098+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:29.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:28.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:29.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:28.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:28.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:28.776+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:29.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:29.145+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:29.145+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:29.145+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:29.146+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:29.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:29.282+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:29.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:29.282+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:29.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:29.282+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:29.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:29.283+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:29.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:29.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:29.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:29.775+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:30.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:30.133+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:30.133+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:30.133+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:30.134+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:30.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:30.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:30.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:30.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:30.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:30.297+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:30.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:30.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:30.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:30.800+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:31.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:31.091+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:31.091+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:31.091+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:31.092+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:31.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:31.284+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:31.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:31.284+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:31.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:31.284+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:31.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:31.284+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:32.076 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:31.773+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:32.076 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:31.773+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:32.076 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:31.773+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:32.077 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:31.774+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:32.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:16:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:16:32.189+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 21247 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:16:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:32.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:32.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:32.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:32.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:32.075+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:32.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:32.292+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:32.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:32.292+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:32.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:32.292+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:32.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:32.292+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:32.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:32.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:32.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:32.756+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:33.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:33.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:33.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:33.079+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:33.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:33.318+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:33.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:33.318+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:33.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:33.318+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:33.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:33.319+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:34.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:33.764+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:33.764+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:33.764+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:33.764+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:34.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:34.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:34.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:34.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:34.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:34.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:34.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:34.331+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:34.128+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:34.128+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:34.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:34.128+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:34.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:34.129+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:35.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:34.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:34.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:34.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:34.775+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:35.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:35.102+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:35.102+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:35.102+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:35.102+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:35.336+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:35.336+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:35.336+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:35.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:35.336+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:36.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:35.733+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:36.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:35.733+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:35.733+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:35.734+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:36.387 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:36.130+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:36.387 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:36.130+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:36.387 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:36.130+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:36.387 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:36.131+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:36.721 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:36.385+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:36.721 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:36.385+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:36.721 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:36.385+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:36.722 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:36.386+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:37.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:36.719+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:37.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:36.719+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:36.719+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:36.720+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:37.410 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:16:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:16:37.190+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 21299 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:16:37.410 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:37.101+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:37.411 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:37.101+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:37.411 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:37.101+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:37.411 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:37.102+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:37.745 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:37.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:37.745 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:37.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:37.745 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:37.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:37.745 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:37.408+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:37.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:37.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:37.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:37.743+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:38.438 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:38.111+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:38.438 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:38.111+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:38.438 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:38.111+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:38.438 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:38.112+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:38.780 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:38.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:38.780 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:38.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:38.780 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:38.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:38.780 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:38.436+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:39.073 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:38.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:39.074 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:38.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:39.074 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:38.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:39.074 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:38.778+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:39.071+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:39.071+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:39.071+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:39.072+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:39.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:39.440+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:39.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:39.440+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:39.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:39.440+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:39.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:39.441+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:40.032 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:39.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:40.032 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:39.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:40.032 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:39.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:40.032 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:39.771+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:40.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:40.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:40.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:40.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:40.030+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:40.754 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:40.412+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:40.755 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:40.412+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:40.755 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:40.412+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:40.755 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:40.412+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:41.024 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:40.752+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:41.025 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:40.752+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:41.025 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:40.752+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:41.025 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:40.752+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:41.022+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:41.022+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:41.022+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:41.023+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:41.772 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:41.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:41.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:41.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:41.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:41.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:41.773 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:41.460+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:42.061 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:41.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:42.061 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:41.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:42.061 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:41.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:42.062 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:41.771+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:42.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:16:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:16:42.192+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 21358 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:16:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:42.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:42.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:42.060+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:42.060+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:42.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:42.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:42.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:42.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:42.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:42.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:42.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:42.486+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:43.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:42.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:42.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:42.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:42.728+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:43.413 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:43.093+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:43.413 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:43.093+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:43.413 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:43.093+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:43.414 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:43.094+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:43.745 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:43.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:43.745 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:43.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:43.745 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:43.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:43.745 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:43.503+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:44.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:43.743+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:44.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:43.743+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:43.743+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:43.744+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:44.464 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:44.083+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:44.465 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:44.083+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:44.465 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:44.083+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:44.465 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:44.083+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:44.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:44.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:44.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:44.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:44.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:44.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:44.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:44.463+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:45.036 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:44.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:45.036 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:44.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:45.036 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:44.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:45.036 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:44.732+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:45.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:45.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:45.034+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:45.035+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:45.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:45.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:45.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:45.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:45.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:45.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:45.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:45.479+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:46.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:45.780+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:45.780+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:45.780+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:45.781+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:46.447 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:46.084+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:46.447 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:46.084+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:46.447 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:46.084+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:46.447 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:46.085+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:46.825 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:46.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:46.825 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:46.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:46.826 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:46.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:46.826 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:46.446+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:47.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:46.824+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:47.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:46.824+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:46.824+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:46.824+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:47.402 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:16:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:16:47.195+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 21417 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:16:47.402 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:47.116+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:47.403 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:47.116+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:47.403 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:47.116+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:47.403 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:47.116+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:47.401+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:47.401+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:47.401+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:47.402+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:48.089 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:47.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:48.089 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:47.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:48.089 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:47.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:48.089 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:47.830+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:48.367 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:48.087+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:48.367 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:48.087+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:48.367 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:48.087+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:48.367 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:48.088+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:48.823 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:48.365+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:48.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:48.365+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:48.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:48.365+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:48.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:48.366+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:49.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:48.822+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:49.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:48.822+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:48.822+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:48.822+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:49.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:49.091+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:49.091+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:49.091+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:49.092+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:49.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:49.327+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:49.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:49.327+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:49.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:49.327+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:49.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:49.328+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:50.091+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:50.091+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:50.091+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:50.092+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:49.870+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:49.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:49.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:49.871+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:50.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:50.332+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:50.332+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:50.332+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:50.333+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:51.329 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:50.849+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:50.849+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:50.849+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:50.850+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:51.049+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:51.049+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:51.049+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:51.050+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:51.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:51.328+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:51.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:51.328+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:51.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:51.328+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:51.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:51.329+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:52.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:51.804+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:52.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:51.804+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:51.804+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:51.805+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:52.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:16:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:16:52.197+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 21476 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:16:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:52.321+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:52.321+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:52.321+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:52.322+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:52.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:52.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:52.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:52.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:52.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:52.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:52.097+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:52.855+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:52.855+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:52.855+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:53.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:52.856+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:53.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:53.084+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:53.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:53.084+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:53.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:53.084+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:53.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:53.085+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:53.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:53.356+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:53.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:53.356+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:53.651 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:53.356+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:53.651 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:53.356+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:54.072 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:53.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:54.072 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:53.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:54.072 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:53.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:54.072 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:53.866+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:54.070+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:54.070+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:54.070+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:54.071+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:54.624 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:54.370+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:54.625 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:54.370+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:54.625 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:54.370+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:54.625 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:54.371+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:55.070+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:55.070+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:55.070+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:55.070+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:54.903+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:54.903+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:54.903+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:54.904+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:55.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:55.394+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:55.394+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:55.394+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:55.395+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:55.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:55.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:55.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:55.935+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:56.074+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:56.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:56.075+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:56.075+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:56.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:56.352+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:56.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:56.352+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:56.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:56.352+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:56.352+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:57.201 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:56.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:57.201 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:56.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:57.202 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:56.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:57.202 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:56.915+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:57.202 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:57.086+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:57.202 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:57.086+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:57.202 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:57.086+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:57.202 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:57.087+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:57.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:16:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:16:57.200+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 21534 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:16:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:57.332+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:57.332+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:57.332+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:57.333+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:58.077+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:58.077+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:58.077+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:58.077+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:57.916+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:57.916+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:57.916+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:58.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:57.917+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:58.704 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:58.352+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:58.704 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:58.352+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:58.704 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:58.352+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:58.705 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:58.353+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:16:59.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:58.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:58.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:58.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:16:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:58.871+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:59.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:59.105+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:59.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:59.105+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:59.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:59.105+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:16:59.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:16:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:16:59.106+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:16:59.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:59.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:59.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:59.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:59.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:59.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:16:59.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:16:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:16:59.313+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:00.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:00.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:00.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:00.067+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:59.873+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:59.873+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:00.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:59.873+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:00.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:16:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:16:59.874+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:00.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:00.353+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:00.353+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:00.353+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:00.354+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:01.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:00.843+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:00.843+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:00.843+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:00.843+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:01.026+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:01.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:01.026+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:01.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:01.026+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:01.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:01.027+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:01.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:01.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:01.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:01.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:01.378+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:02.204 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:01.845+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:02.205 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:01.845+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:02.205 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:01.845+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:02.205 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:01.846+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:02.205 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:02.026+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:02.205 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:02.026+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:02.206 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:02.026+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:02.206 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:02.027+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:02.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:17:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:17:02.202+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 21593 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:17:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:02.360+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:02.360+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:02.360+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:02.360+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:03.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:03.000+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:03.000+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:03.000+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:03.001+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:02.825+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:02.825+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:03.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:02.825+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:03.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:02.826+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:03.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:03.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:03.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:03.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:03.651 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:03.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:03.651 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:03.396+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:04.032 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:03.857+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:04.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:03.857+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:04.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:03.857+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:04.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:03.858+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:04.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:04.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:04.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:04.032+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:04.371+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:04.371+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:04.371+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:04.371+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:04.995+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:04.995+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:04.995+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:04.996+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:04.905+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:04.905+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:04.905+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:05.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:04.906+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:05.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:05.333+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:05.333+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:05.333+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:05.333+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:05.982+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:05.982+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:05.982+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:05.982+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:05.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:05.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:05.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:05.922+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:06.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:06.336+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:06.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:06.336+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:06.336+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:06.337+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:07.205 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:06.893+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:07.206 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:06.893+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:07.206 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:06.893+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:07.206 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:06.893+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:07.206 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:06.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:07.206 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:06.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:07.207 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:06.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:07.207 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:06.993+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:07.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:17:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:17:07.204+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 21652 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:17:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:07.303+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:07.303+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:07.303+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:07.303+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:08.325 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:07.958+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:08.325 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:07.958+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:08.326 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:07.958+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:08.326 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:07.959+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:08.326 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:07.858+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:08.326 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:07.858+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:08.326 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:07.858+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:08.326 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:07.859+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:08.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:08.324+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:08.324+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:08.324+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:08.324+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:09.308 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:08.865+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:09.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:08.865+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:09.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:08.865+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:09.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:08.865+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:09.309 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:08.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:09.309 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:08.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:09.309 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:08.991+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:09.310 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:08.992+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:09.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:09.306+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:09.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:09.306+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:09.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:09.306+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:09.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:09.307+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:10.301 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:09.897+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:10.301 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:09.897+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:10.301 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:09.897+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:10.301 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:09.897+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:10.302 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:09.947+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:10.302 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:09.947+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:10.302 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:09.947+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:10.302 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:09.948+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:10.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:10.297+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:10.297+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:10.297+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:10.298+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:11.318 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:10.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:11.319 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:10.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:11.319 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:10.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:11.319 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:10.948+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:11.319 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:10.967+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:11.319 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:10.967+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:11.319 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:10.967+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:11.320 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:10.968+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:11.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:11.317+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:11.317+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:11.317+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:11.317+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:12.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:17:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:17:12.206+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 21711 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:17:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:11.968+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:11.968+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:11.968+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:11.969+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:12.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:12.003+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:12.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:12.003+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:12.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:12.003+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:12.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:12.004+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:12.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:12.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:12.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:12.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:12.342+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:13.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:13.014+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:13.014+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:13.014+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:13.015+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:13.003+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:13.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:13.003+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:13.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:13.003+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:13.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:13.004+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:13.651 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:13.338+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:13.651 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:13.338+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:13.651 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:13.338+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:13.651 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:13.339+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:14.295 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:13.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:14.296 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:13.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:14.296 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:13.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:14.296 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:13.956+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:14.296 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:13.992+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:14.296 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:13.992+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:14.296 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:13.992+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:14.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:13.993+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:14.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:14.293+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:14.293+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:14.293+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:14.293+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:15.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:15.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:15.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:15.003+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:15.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:14.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:15.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:14.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:15.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:14.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:15.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:14.932+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:15.809 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:15.334+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:15.809 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:15.334+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:15.810 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:15.334+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:15.810 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:15.334+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:16.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:15.937+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:15.937+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:15.937+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:15.938+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:15.976+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:15.976+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:15.976+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:16.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:15.977+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:16.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:16.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:16.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:16.312+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:17.210 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:17.010+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:17.211 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:17.010+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:17.211 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:17.010+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:17.211 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:17.010+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:17.211 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:16.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:17.211 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:16.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:17.212 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:16.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:17.212 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:16.906+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:17.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:17:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:17:17.207+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 21769 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:17:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:17.276+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:17.276+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:17.276+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:17.277+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:18.287 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:18.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:18.287 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:18.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:18.287 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:18.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:18.287 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:18.038+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:18.287 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:17.861+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:18.288 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:17.861+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:18.288 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:17.861+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:18.288 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:17.862+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:18.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:18.284+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:18.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:18.284+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:18.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:18.284+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:18.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:18.285+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:19.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:18.823+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:18.823+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:18.823+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:18.824+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:19.032+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:19.032+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:19.032+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:19.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:19.033+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:19.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:19.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:19.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:19.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:19.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:19.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:19.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:19.330+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:20.047 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:19.779+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:20.047 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:19.779+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:20.048 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:19.779+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:20.048 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:19.779+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:20.303 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:20.045+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:20.304 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:20.045+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:20.304 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:20.045+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:20.304 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:20.047+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:20.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:20.301+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:20.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:20.302+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:20.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:20.302+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:20.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:20.302+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:21.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:21.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:21.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:21.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:21.017+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:20.781+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:21.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:20.781+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:21.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:20.781+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:21.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:20.781+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:21.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:21.273+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:21.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:21.273+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:21.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:21.273+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:21.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:21.273+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:22.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:22.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:22.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:22.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:22.039+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:21.824+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:21.824+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:21.824+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:22.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:21.825+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:22.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:17:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:17:22.210+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 21827 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:17:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:22.228+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:22.228+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:22.228+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:22.229+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:23.259 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:23.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:23.259 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:23.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:23.259 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:23.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:23.259 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:23.007+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:23.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:22.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:23.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:22.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:23.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:22.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:23.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:22.853+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:23.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:23.257+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:23.257+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:23.257+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:23.258+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:24.035 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:23.872+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:24.036 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:23.872+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:24.036 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:23.872+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:24.036 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:23.873+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:24.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:24.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:24.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:24.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:24.035+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:24.264+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:24.264+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:24.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:24.264+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:24.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:24.264+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:25.284 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:24.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:25.284 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:24.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:25.285 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:24.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:25.285 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:24.996+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:25.285 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:24.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:25.285 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:24.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:25.285 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:24.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:25.286 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:24.872+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:25.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:25.283+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:25.283+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:25.283+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:25.283+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:26.267 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:25.970+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:26.268 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:25.970+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:26.268 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:25.970+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:26.268 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:25.971+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:26.268 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:25.849+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:26.268 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:25.849+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:26.269 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:25.849+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:26.269 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:25.849+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:26.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:26.266+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:26.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:26.266+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:26.266+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:26.267+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:27.215 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:26.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:27.215 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:26.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:27.215 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:26.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:27.215 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:26.994+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:27.216 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:26.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:27.216 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:26.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:27.216 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:26.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:27.216 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:26.872+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:27.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:17:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:17:27.212+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 21887 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:17:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:27.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:27.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:27.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:27.314+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:28.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:28.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:28.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:28.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:28.272 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:28.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:28.272 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:28.020+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:28.272 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:27.858+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:28.272 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:27.858+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:28.272 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:27.858+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:28.272 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:27.858+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:28.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:28.270+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:28.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:28.270+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:28.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:28.270+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:28.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:28.270+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:29.176 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:28.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:29.176 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:28.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:29.176 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:28.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:29.177 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:28.981+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:29.177 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:28.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:29.177 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:28.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:29.177 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:28.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:29.177 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:28.872+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:29.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:29.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:29.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:29.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:29.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:29.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:29.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:29.312+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:29.957+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:29.957+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:29.957+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:29.957+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:29.898+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:30.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:29.898+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:30.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:29.898+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:30.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:29.899+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:30.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:30.360+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:30.360+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:30.360+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:30.361+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:30.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:30.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:30.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:30.913+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:30.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:30.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:30.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:31.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:30.941+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:31.681 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:31.385+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:31.681 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:31.385+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:31.682 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:31.385+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:31.682 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:31.385+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:32.217 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:31.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:32.217 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:31.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:32.217 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:31.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:32.217 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:31.921+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:32.218 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:31.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:32.218 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:31.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:32.218 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:31.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:32.218 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:31.923+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:32.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:17:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:17:32.215+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 21945 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:17:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:32.432+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:32.432+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:32.432+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:32.433+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:33.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:32.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:32.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:32.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:32.903+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:32.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:32.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:32.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:33.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:32.876+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:33.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:33.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:33.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:33.442+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:34.249 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:33.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:34.249 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:33.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:34.249 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:33.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:34.249 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:33.902+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:34.250 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:33.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:34.250 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:33.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:34.250 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:33.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:34.250 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:33.919+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:34.561 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:34.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:34.561 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:34.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:34.561 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:34.446+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:34.561 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:34.447+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:35.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:34.923+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:35.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:34.923+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:35.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:34.923+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:35.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:34.924+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:35.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:34.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:35.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:34.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:35.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:34.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:35.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:34.895+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:35.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:35.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:35.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:35.482+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:36.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:35.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:35.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:35.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:35.922+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:35.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:35.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:35.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:36.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:35.928+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:36.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:36.453+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:36.453+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:36.453+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:36.453+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:37.219 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:36.965+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:37.219 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:36.965+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:37.219 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:36.965+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:37.219 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:36.966+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:37.220 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:36.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:37.220 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:36.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:37.220 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:36.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:37.220 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:36.921+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:37.491 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:17:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:17:37.217+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22003 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:17:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:37.490+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:37.490+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:37.490+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:37.490+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:38.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:38.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:38.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:38.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:38.015+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:38.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:37.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:38.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:37.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:38.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:37.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:38.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:37.879+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:38.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:38.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:38.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:38.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:38.497+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:39.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:39.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:39.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:39.064+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:38.851+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:38.851+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:39.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:38.851+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:39.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:38.852+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:39.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:39.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:39.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:39.526+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:40.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:40.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:40.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:40.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:40.038+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:39.900+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:39.900+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:39.900+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:40.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:39.900+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:40.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:40.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:40.525+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:40.526+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:41.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:41.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:41.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:41.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:41.041+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:40.854+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:40.854+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:41.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:40.854+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:41.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:40.855+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:41.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:41.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:41.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:41.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:41.552+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:42.053 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T04:17:42.050+0000 7f4da1155640 0 monclient(hunting): authenticate timed out after 300 2024-09-10T04:17:42.053 INFO:teuthology.orchestra.run.smithi112.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-09-10T04:17:42.221 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:42.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:42.221 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:42.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:42.221 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:42.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:42.221 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:42.081+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:42.222 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:41.887+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:42.222 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:41.887+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:42.222 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:41.887+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:42.222 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:41.888+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:42.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:17:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:17:42.219+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22062 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:17:42.860 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:42.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:42.861 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:42.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:42.861 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:42.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:42.861 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:42.592+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:42.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T04:17:42.862 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-09-10T04:17:42.863 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T04:17:42.863 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T04:17:42.880 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-10T04:17:42.880 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T04:17:42.936 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd blocklist ls 2024-09-10T04:17:43.242 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T04:17:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:42.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:42.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:42.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:42.923+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:43.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:43.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:43.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:43.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:43.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:43.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:43.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:43.065+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:43.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:43.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:43.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:43.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:43.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:43.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:43.957 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:43.627+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:43.953+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:43.953+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:43.953+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:43.955+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:44.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:44.060+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:44.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:44.060+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:44.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:44.060+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:44.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:44.061+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:44.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:44.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:44.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:44.654+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:44.655+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:44.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:44.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:44.964+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:44.965+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:45.026+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:45.026+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:45.026+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:45.027+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:46.012 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:45.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:46.012 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:45.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:46.012 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:45.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:46.012 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:45.628+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:46.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:46.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:46.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:46.035+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:46.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:46.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:46.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:46.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:46.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:46.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:46.011+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:46.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:46.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:46.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:46.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:46.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:46.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:46.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:46.635+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:47.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:17:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:17:47.221+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22121 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:17:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:47.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:47.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:47.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:47.035+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:46.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:47.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:46.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:47.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:46.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:47.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:46.990+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:47.954 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:47.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:47.954 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:47.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:47.954 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:47.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:47.954 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:47.638+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:48.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:48.075+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:48.075+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:48.075+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:48.076+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:47.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:47.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:47.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:48.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:47.953+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:48.909 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:48.652+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:48.909 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:48.652+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:48.909 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:48.652+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:48.909 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:48.653+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:49.096+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:49.096+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:49.096+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:49.097+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:48.907+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:48.907+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:49.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:48.907+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:49.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:48.908+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:49.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:49.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:49.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:49.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:49.694+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:50.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:50.054+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:50.054+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:50.054+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:50.055+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:49.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:49.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:49.917+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:50.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:49.918+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:51.069 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:50.730+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:51.069 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:50.730+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:51.069 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:50.730+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:51.069 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:50.730+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:51.069 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:50.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:51.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:50.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:51.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:50.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:51.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:50.907+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:51.066+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:51.066+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:51.066+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:51.067+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:52.033 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:51.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:52.033 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:51.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:52.033 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:51.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:52.033 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:51.882+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:52.034 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:51.701+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:52.034 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:51.701+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:52.034 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:51.701+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:52.034 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:51.702+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:52.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:17:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:17:52.222+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22180 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:17:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:52.030+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:52.030+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:52.030+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:52.031+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:53.075 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:52.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:53.075 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:52.665+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:53.076 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:52.665+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:53.076 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:52.665+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:53.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:52.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:53.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:52.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:53.077 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:52.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:53.077 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:52.905+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:53.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:53.073+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:53.073+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:53.073+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:53.074+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:54.079 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:53.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:54.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:53.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:54.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:53.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:54.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:53.685+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:54.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:53.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:54.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:53.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:53.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:53.856+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:54.077+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:54.077+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:54.077+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:54.078+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:54.872 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:54.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:54.872 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:54.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:54.872 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:54.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:54.872 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:54.683+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:55.096+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:55.096+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:55.096+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:55.097+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:54.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:54.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:54.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:54.870+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:56.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:55.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:55.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:55.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:55.714+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:55.883+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:55.883+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:55.883+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:56.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:55.883+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:56.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:56.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:56.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:56.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:56.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:56.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:56.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:56.122+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:57.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:56.689+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:57.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:56.689+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:57.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:56.689+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:57.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:56.689+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:56.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:56.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:57.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:56.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:57.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:56.920+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:57.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:17:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:17:57.226+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22238 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:17:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:57.141+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:57.141+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:57.141+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:57.141+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:58.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:57.884+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:57.884+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:57.884+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:57.884+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:57.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:57.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:58.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:57.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:58.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:57.687+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:58.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:58.187+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:58.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:58.187+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:58.187+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:58.188+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:59.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:58.923+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:58.923+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:58.923+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:17:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:58.924+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:58.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:58.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:58.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:59.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:58.700+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:17:59.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:59.214+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:59.214+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:59.214+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:17:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:17:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:17:59.215+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:17:59.943 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:59.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:59.943 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:59.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:59.943 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:59.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:17:59.944 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:17:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:17:59.715+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:00.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:00.172+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:00.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:00.172+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:00.172+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:00.173+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:59.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:59.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:59.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:17:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:17:59.943+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:01.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:00.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:00.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:00.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:00.898+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:01.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:00.749+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:01.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:00.749+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:01.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:00.749+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:01.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:00.750+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:01.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:01.156+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:01.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:01.156+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:01.156+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:01.157+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:02.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:01.867+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:01.867+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:01.867+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:01.868+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:01.768+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:01.768+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:01.768+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:01.769+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:02.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:18:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:18:02.228+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22297 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:18:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:02.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:02.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:02.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:02.163+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:03.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:02.831+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:02.831+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:02.831+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:02.831+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:02.757+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:02.757+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:03.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:02.757+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:03.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:02.758+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:03.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:03.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:03.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:03.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:03.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:03.182+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:04.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:03.824+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:03.824+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:03.824+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:03.825+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:03.775+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:03.775+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:03.775+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:03.776+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:04.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:04.177+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:04.177+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:04.177+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:04.178+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:05.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:04.819+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:04.819+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:04.819+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:04.820+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:05.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:04.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:05.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:04.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:05.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:04.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:05.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:04.844+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:05.472 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:05.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:05.472 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:05.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:05.473 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:05.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:05.473 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:05.166+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:06.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:05.831+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:05.831+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:05.831+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:05.832+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:05.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:05.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:06.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:05.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:06.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:05.793+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:06.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:06.177+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:06.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:06.177+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:06.177+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:06.178+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:07.149 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:06.833+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:07.149 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:06.833+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:07.149 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:06.833+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:07.149 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:06.833+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:07.150 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:06.849+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:07.150 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:06.849+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:07.150 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:06.849+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:07.150 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:06.849+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:07.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:18:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:18:07.230+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22356 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:18:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:07.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:07.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:07.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:07.148+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:07.871 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:07.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:07.872 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:07.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:07.872 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:07.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:08.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:07.870+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:08.100+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:08.100+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:08.100+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:08.101+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:07.883+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:08.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:07.883+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:08.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:07.883+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:08.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:07.883+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:09.130 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:08.915+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:09.130 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:08.915+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:09.130 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:08.915+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:09.130 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:08.916+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:09.131 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:08.877+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:09.131 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:08.877+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:09.131 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:08.877+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:09.131 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:08.879+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:09.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:09.128+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:09.128+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:09.128+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:09.129+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:10.149 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:09.833+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:10.149 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:09.833+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:10.149 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:09.833+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:10.149 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:09.833+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:10.150 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:09.943+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:10.150 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:09.943+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:10.150 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:09.943+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:10.150 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:09.944+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:10.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:10.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:10.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:10.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:10.148+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:11.172 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:10.953+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:11.172 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:10.953+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:11.172 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:10.953+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:11.172 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:10.953+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:11.173 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:10.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:11.173 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:10.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:11.173 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:10.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:11.173 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:10.846+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:11.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:11.170+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:11.170+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:11.170+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:11.171+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:12.181 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:11.981+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:12.181 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:11.981+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:12.182 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:11.981+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:12.182 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:11.981+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:12.182 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:11.887+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:12.182 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:11.887+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:12.182 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:11.887+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:12.183 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:11.888+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:12.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:18:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:18:12.233+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22415 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:18:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:12.180+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:12.180+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:12.180+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:12.181+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:13.196 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:13.009+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:13.197 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:13.009+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:13.197 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:13.009+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:13.197 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:13.010+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:13.197 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:12.868+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:13.197 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:12.868+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:13.197 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:12.868+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:13.198 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:12.869+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:13.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:13.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:13.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:13.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:13.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:13.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:13.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:13.194+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:13.968 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:13.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:13.968 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:13.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:13.968 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:13.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:13.969 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:13.869+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:14.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:13.966+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:14.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:13.966+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:14.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:13.966+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:14.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:13.967+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:14.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:14.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:14.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:14.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:14.190+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:15.148 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:14.865+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:15.148 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:14.865+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:15.148 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:14.865+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:15.149 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:14.866+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:15.149 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:14.948+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:15.149 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:14.948+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:15.149 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:14.948+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:15.149 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:14.949+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:15.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:15.146+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:15.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:15.146+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:15.146+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:15.147+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:16.184 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:15.992+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:16.184 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:15.992+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:16.185 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:15.992+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:16.185 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:15.993+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:16.185 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:15.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:16.185 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:15.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:16.185 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:15.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:16.185 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:15.880+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:16.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:16.183+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:16.183+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:16.183+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:16.183+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:17.191 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:16.954+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:17.191 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:16.954+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:17.191 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:16.954+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:17.192 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:16.955+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:17.192 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:16.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:17.192 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:16.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:17.192 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:16.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:17.192 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:16.890+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:17.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:18:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:18:17.235+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22473 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:18:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:17.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:17.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:17.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:17.190+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:18.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:17.921+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:18.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:17.921+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:18.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:17.921+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:18.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:17.922+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:17.983+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:17.983+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:18.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:17.983+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:18.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:17.984+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:18.355 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:18.226+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:18.355 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:18.226+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:18.356 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:18.226+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:18.356 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:18.226+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:19.260 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:18.938+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:19.261 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:18.938+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:19.261 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:18.938+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:19.261 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:18.939+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:19.261 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:18.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:19.261 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:18.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:19.261 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:18.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:19.261 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:18.955+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:19.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:19.259+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:19.259+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:19.259+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:19.260+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:20.217 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:19.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:20.218 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:19.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:20.218 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:19.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:20.218 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:19.909+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:20.218 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:19.976+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:20.218 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:19.976+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:20.219 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:19.976+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:20.219 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:19.977+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:20.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:20.216+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:20.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:20.216+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:20.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:20.216+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:20.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:20.216+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:21.267 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:20.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:21.267 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:20.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:21.268 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:20.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:21.268 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:20.891+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:21.268 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:21.013+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:21.268 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:21.013+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:21.268 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:21.013+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:21.268 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:21.013+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:21.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:21.265+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:21.265+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:21.265+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:21.266+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:22.239 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:22.020+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:22.239 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:22.020+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:22.239 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:22.020+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:22.239 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:22.021+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:22.240 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:21.865+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:22.240 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:21.865+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:22.240 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:21.865+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:22.240 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:21.865+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:22.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:18:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:18:22.237+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22532 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:18:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:22.244+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:22.244+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:22.244+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:22.245+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:23.285 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:22.859+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:23.286 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:22.859+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:23.286 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:22.859+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:23.286 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:22.860+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:23.286 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:23.041+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:23.286 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:23.041+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:23.287 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:23.041+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:23.287 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:23.042+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:23.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:23.284+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:23.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:23.284+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:23.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:23.284+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:23.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:23.285+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:24.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:23.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:24.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:23.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:24.071 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:23.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:24.071 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:23.900+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:24.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:24.302+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:24.302+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:24.302+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:24.303+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:24.069+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:24.069+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:24.069+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:24.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:24.070+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:25.306 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:25.040+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:25.306 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:25.040+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:25.306 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:25.040+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:25.306 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:25.040+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:25.307 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:24.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:25.307 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:24.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:25.307 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:24.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:25.307 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:24.879+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:25.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:25.304+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:25.304+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:25.304+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:25.305+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:26.290 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:25.921+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:26.291 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:25.921+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:26.291 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:25.921+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:26.291 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:25.922+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:26.291 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:26.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:26.291 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:26.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:26.291 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:26.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:26.292 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:26.089+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:26.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:26.289+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:26.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:26.289+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:26.289+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:26.289+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:27.241 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:27.103+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:27.241 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:27.103+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:27.241 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:27.103+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:27.242 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:27.104+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:27.242 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:26.945+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:27.242 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:26.945+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:27.242 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:26.945+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:27.242 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:26.946+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:27.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:18:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:18:27.239+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22591 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:18:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:27.294+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:27.294+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:27.294+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:27.295+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:28.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:27.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:28.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:27.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:28.323 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:27.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:28.323 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:27.943+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:28.323 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:28.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:28.323 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:28.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:28.323 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:28.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:28.323 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:28.151+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:28.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:28.319+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:28.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:28.319+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:28.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:28.319+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:28.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:28.320+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:29.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:29.113+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:29.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:29.113+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:29.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:29.113+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:29.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:29.113+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:29.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:28.959+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:29.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:28.959+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:29.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:28.959+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:29.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:28.960+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:29.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:29.330+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:29.330+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:29.330+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:29.330+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:30.202 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:30.081+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:30.202 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:30.081+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:30.202 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:30.081+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:30.202 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:30.081+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:30.203 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:29.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:30.203 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:29.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:30.203 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:29.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:30.203 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:29.913+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:30.354+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:30.354+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:30.354+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:30.356+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:31.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:31.100+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:31.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:31.100+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:31.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:31.100+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:31.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:31.101+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:30.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:30.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:31.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:30.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:31.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:30.897+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:31.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:31.386+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:31.386+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:31.386+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:31.387+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:32.243 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:32.068+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:32.243 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:32.068+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:32.244 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:32.068+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:32.244 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:32.069+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:32.244 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:31.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:32.244 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:31.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:32.244 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:31.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:32.244 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:31.928+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:32.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:18:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:18:32.241+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22650 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:18:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:32.342+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:32.342+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:32.342+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:32.342+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:33.312 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:32.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:33.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:32.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:33.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:32.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:33.313 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:32.940+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:33.313 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:33.078+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:33.314 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:33.078+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:33.314 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:33.078+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:33.314 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:33.079+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:33.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:33.310+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:33.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:33.310+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:33.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:33.310+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:33.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:33.310+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:34.272 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:33.947+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:34.272 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:33.947+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:34.273 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:33.947+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:34.273 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:33.948+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:34.273 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:34.095+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:34.273 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:34.095+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:34.273 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:34.095+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:34.273 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:34.095+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:34.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:34.271+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:34.271+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:34.271+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:34.271+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:35.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:34.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:35.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:34.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:35.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:34.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:35.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:34.928+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:35.256 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:35.085+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:35.256 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:35.085+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:35.256 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:35.085+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:35.256 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:35.085+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:35.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:35.253+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:35.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:35.253+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:35.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:35.253+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:35.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:35.254+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:36.257 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:35.947+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:36.257 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:35.947+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:36.257 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:35.947+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:36.257 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:35.948+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:36.258 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:36.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:36.258 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:36.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:36.258 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:36.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:36.258 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:36.133+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:36.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:36.255+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:36.255+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:36.255+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:36.256+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:37.237 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:37.137+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:37.237 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:37.137+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:37.237 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:37.137+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:37.238 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:37.137+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:37.238 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:36.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:37.238 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:36.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:37.238 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:36.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:37.238 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:36.974+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:37.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:18:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:18:37.244+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22708 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:18:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:37.235+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:37.235+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:37.235+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:37.236+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:38.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:38.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:38.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:38.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:38.190+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:38.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:38.091+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:38.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:38.091+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:38.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:38.091+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:38.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:38.092+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:38.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:37.974+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:38.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:37.974+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:38.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:37.974+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:38.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:37.975+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:38.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:38.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:38.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:38.958+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:39.146+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:39.146+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:39.146+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:39.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:39.147+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:39.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:39.124+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:39.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:39.124+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:39.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:39.124+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:39.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:39.125+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:40.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:40.142+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:40.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:40.142+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:40.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:40.142+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:40.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:40.143+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:40.101+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:40.101+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:40.101+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:40.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:40.102+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:40.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:39.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:40.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:39.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:40.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:39.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:40.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:39.943+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:41.099+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:41.099+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:41.099+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:41.100+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:41.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:40.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:41.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:40.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:41.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:40.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:41.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:40.961+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:41.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:41.135+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:41.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:41.135+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:41.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:41.135+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:41.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:41.136+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:42.248 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:42.112+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:42.248 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:42.112+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:42.248 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:42.112+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:42.248 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:42.113+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:42.249 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:41.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:42.249 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:41.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:42.249 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:41.955+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:42.249 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:41.956+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:42.249 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:42.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:42.250 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:42.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:42.250 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:42.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:42.250 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:42.090+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:42.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:18:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:18:42.246+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22767 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:18:43.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:43.122+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:43.122+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:43.122+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:43.123+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:42.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:42.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:43.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:42.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:43.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:42.923+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:43.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:43.058+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:43.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:43.058+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:43.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:43.058+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:43.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:43.059+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:44.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:43.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:44.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:43.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:44.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:43.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:44.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:43.883+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:44.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:44.104+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:44.104+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:44.104+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:44.105+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:44.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:44.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:44.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:44.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:44.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:44.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:44.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:44.084+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:45.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:44.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:44.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:44.931+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:44.932+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:45.129+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:45.129+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:45.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:45.129+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:45.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:45.130+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:45.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:45.065+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:45.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:45.065+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:45.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:45.065+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:45.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:45.066+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:46.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:46.110+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:46.110+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:46.110+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:46.111+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:45.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:45.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:46.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:45.902+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:46.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:45.903+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:46.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:46.044+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:46.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:46.044+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:46.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:46.044+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:46.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:46.045+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:47.248 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:47.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:47.248 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:47.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:47.249 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:47.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:47.249 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:47.075+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:47.249 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:47.012+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:47.249 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:47.012+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:47.249 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:47.012+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:47.249 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:47.013+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:47.250 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:46.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:47.250 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:46.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:47.250 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:46.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:47.250 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:46.863+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:47.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:18:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:18:47.249+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22825 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:18:48.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:47.907+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:47.907+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:47.907+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:47.908+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:48.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:48.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:48.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:48.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:48.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:48.039+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:48.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:48.043+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:48.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:48.043+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:48.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:48.043+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:48.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:48.043+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:49.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:49.062+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:49.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:49.062+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:49.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:49.062+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:49.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:49.063+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:48.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:48.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:48.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:48.900+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:49.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:49.058+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:49.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:49.058+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:49.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:49.058+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:49.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:49.059+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:50.095 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:50.057+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:50.095 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:50.057+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:50.096 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:50.057+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:50.096 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:50.058+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:50.096 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:49.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:50.096 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:49.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:50.096 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:49.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:50.096 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:49.951+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:50.531 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:50.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:50.531 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:50.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:50.531 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:50.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:50.532 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:50.095+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:51.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:51.106+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:51.106+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:51.106+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:51.107+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:51.126+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:51.126+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:51.126+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:51.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:51.127+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:51.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:50.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:51.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:50.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:51.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:50.906+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:51.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:50.906+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:52.253 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:52.091+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:52.253 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:52.091+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:52.253 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:52.091+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:52.253 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:52.092+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:52.254 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:52.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:52.254 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:52.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:52.254 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:52.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:52.254 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:52.150+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:52.254 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:51.945+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:52.254 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:51.945+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:52.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:51.945+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:52.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:51.945+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:52.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:18:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:18:52.251+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22884 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:18:53.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:52.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:52.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:52.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:52.944+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:53.158+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:53.158+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:53.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:53.158+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:53.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:53.159+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:53.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:53.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:53.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:53.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:53.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:53.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:53.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:53.074+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:54.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:53.903+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:53.903+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:53.903+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:53.903+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:54.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:54.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:54.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:54.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:54.163+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:54.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:54.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:54.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:54.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:54.089+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:55.153 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:54.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:55.153 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:54.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:55.153 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:54.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:55.153 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:54.891+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:55.153 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:55.099+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:55.154 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:55.099+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:55.154 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:55.099+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:55.154 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:55.100+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:55.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:55.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:55.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:55.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:55.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:55.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:55.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:55.152+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:56.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:56.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:56.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:56.156+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:56.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:56.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:56.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:56.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:56.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:56.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:56.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:56.088+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:56.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:55.935+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:56.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:55.935+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:56.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:55.935+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:56.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:55.936+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:57.255 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:57.123+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:57.255 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:57.123+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:57.256 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:57.123+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:57.256 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:57.124+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:57.256 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:57.193+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:57.256 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:57.193+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:57.256 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:57.193+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:57.257 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:57.194+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:57.257 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:56.944+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:57.257 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:56.944+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:57.257 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:56.944+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:57.257 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:56.945+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:57.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:18:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:18:57.253+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 22942 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:18:58.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:58.179+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:58.179+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:58.179+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:58.180+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:58.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:58.085+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:58.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:58.085+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:58.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:58.085+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:58.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:58.086+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:18:58.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:57.987+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:58.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:57.987+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:58.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:57.987+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:58.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:57.987+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:59.023+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:59.023+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:59.023+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:18:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:59.023+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:59.225+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:59.225+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:59.225+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:18:59.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:18:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:18:59.225+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:18:59.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:59.067+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:59.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:59.067+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:59.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:59.067+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:18:59.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:18:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:18:59.068+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:59.998+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:59.998+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:59.998+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:18:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:18:59.998+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:00.237+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:00.237+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:00.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:00.237+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:00.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:00.238+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:00.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:00.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:00.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:00.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:00.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:00.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:00.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:00.088+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:01.046+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:01.046+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:01.047+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:01.047+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:01.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:01.130+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:01.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:01.130+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:01.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:01.130+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:01.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:01.130+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:01.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:01.281+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:01.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:01.281+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:01.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:01.281+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:01.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:01.281+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:02.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:19:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:19:02.256+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23001 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:19:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:02.006+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:02.006+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:02.006+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:02.007+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:02.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:02.241+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:02.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:02.241+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:02.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:02.241+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:02.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:02.241+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:02.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:02.141+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:02.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:02.141+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:02.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:02.141+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:02.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:02.141+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:03.239 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:02.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:03.240 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:02.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:03.240 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:02.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:03.240 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:02.967+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:03.240 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:03.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:03.240 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:03.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:03.241 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:03.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:03.241 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:03.120+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:03.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:03.237+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:03.237+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:03.237+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:03.238+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:04.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:04.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:04.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:04.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:04.009+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:04.190+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:04.190+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:04.190+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:04.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:04.190+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:04.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:04.163+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:04.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:04.163+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:04.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:04.163+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:04.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:04.163+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:05.056+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:05.056+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:05.056+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:05.057+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:05.231+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:05.231+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:05.231+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:05.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:05.232+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:05.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:05.144+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:05.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:05.144+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:05.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:05.144+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:05.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:05.145+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:06.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:06.201+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:06.201+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:06.201+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:06.202+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:06.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:06.165+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:06.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:06.165+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:06.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:06.165+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:06.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:06.166+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:06.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:06.057+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:06.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:06.057+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:06.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:06.057+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:06.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:06.058+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:07.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:19:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:19:07.257+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23060 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:19:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:07.216+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:07.216+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:07.216+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:07.217+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:07.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:07.089+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:07.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:07.089+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:07.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:07.089+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:07.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:07.090+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:07.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:07.139+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:07.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:07.139+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:07.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:07.139+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:07.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:07.140+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:08.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:08.126+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:08.126+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:08.126+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:08.127+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:08.183+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:08.183+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:08.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:08.183+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:08.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:08.184+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:08.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:08.181+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:08.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:08.181+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:08.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:08.181+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:08.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:08.182+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:09.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:09.233+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:09.233+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:09.233+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:09.234+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:09.165+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:09.165+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:09.165+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:09.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:09.166+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:09.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:09.171+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:09.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:09.171+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:09.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:09.171+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:09.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:09.171+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:10.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:10.200+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:10.200+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:10.200+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:10.201+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:10.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:10.167+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:10.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:10.167+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:10.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:10.167+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:10.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:10.168+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:10.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:10.236+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:10.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:10.236+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:10.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:10.236+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:10.583 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:10.237+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:11.231+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:11.231+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:11.231+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:11.232+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:11.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:11.175+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:11.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:11.175+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:11.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:11.175+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:11.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:11.175+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:11.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:11.198+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:11.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:11.198+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:11.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:11.198+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:11.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:11.198+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:12.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:19:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:19:12.260+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23119 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:19:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:12.283+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:12.283+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:12.283+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:12.283+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:12.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:12.163+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:12.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:12.163+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:12.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:12.163+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:12.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:12.164+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:12.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:12.184+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:12.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:12.184+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:12.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:12.184+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:12.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:12.185+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:13.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:13.270+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:13.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:13.270+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:13.270+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:13.270+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:13.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:13.172+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:13.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:13.172+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:13.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:13.172+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:13.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:13.173+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:13.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:13.167+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:13.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:13.167+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:13.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:13.167+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:13.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:13.167+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:14.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:14.237+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:14.237+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:14.237+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:14.238+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:14.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:14.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:14.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:14.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:14.213+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:14.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:14.214+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:14.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:14.201+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:14.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:14.201+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:14.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:14.201+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:14.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:14.202+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:15.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:15.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:15.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:15.189+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:15.189+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:15.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:15.217+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:15.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:15.217+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:15.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:15.217+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:15.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:15.217+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:15.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:15.216+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:15.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:15.216+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:15.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:15.216+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:15.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:15.217+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:16.180 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:16.164+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:16.180 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:16.164+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:16.180 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:16.164+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:16.180 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:16.165+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:16.181 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:16.176+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:16.181 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:16.176+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:16.181 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:16.176+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:16.181 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:16.176+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:16.181 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:16.177+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:16.181 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:16.177+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:16.182 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:16.177+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:16.182 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:16.178+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:17.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:17.190+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:17.190+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:17.190+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:17.191+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:17.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:17.131+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:17.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:17.131+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:17.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:17.131+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:17.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:17.132+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:17.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:17.200+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:17.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:17.200+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:17.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:17.200+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:17.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:17.201+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:17.583 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:19:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:19:17.262+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23177 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:19:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:18.165+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:18.165+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:18.165+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:18.166+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:18.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:18.136+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:18.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:18.136+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:18.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:18.136+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:18.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:18.137+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:18.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:18.233+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:18.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:18.233+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:18.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:18.233+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:18.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:18.233+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:19.482 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:19.199+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:19.483 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:19.199+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:19.483 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:19.199+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:19.483 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:19.200+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:19.483 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:19.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:19.483 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:19.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:19.483 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:19.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:19.484 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:19.121+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:19.484 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:19.158+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:19.484 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:19.158+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:19.484 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:19.158+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:19.484 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:19.158+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:20.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:20.075+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:20.075+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:20.075+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:20.076+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:20.249+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:20.249+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:20.249+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:20.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:20.250+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:20.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:20.174+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:20.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:20.174+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:20.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:20.174+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:20.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:20.174+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:21.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:21.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:21.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:21.150+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:21.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:21.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:21.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:21.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:21.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:21.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:21.240+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:21.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:21.044+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:21.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:21.044+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:21.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:21.044+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:21.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:21.045+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:22.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:22.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:22.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:22.036+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:22.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:22.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:22.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:22.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:22.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:22.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:22.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:22.260+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:22.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:22.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:22.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:22.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:22.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:22.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:22.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:22.109+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:22.333 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:19:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:19:22.265+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23236 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:19:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:23.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:23.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:23.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:23.016+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:23.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:23.259+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:23.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:23.259+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:23.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:23.259+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:23.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:23.260+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:23.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:23.151+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:23.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:23.151+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:23.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:23.151+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:23.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:23.152+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:24.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:24.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:24.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:24.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:24.004+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:24.217+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:24.217+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:24.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:24.217+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:24.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:24.218+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:24.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:24.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:24.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:24.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:24.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:24.108+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:24.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:24.109+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:25.014+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:25.014+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:25.014+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:25.015+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:25.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:25.246+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:25.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:25.246+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:25.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:25.246+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:25.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:25.246+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:25.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:25.113+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:25.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:25.113+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:25.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:25.113+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:25.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:25.114+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:26.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:26.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:26.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:26.060+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:26.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:26.208+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:26.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:26.208+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:26.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:26.208+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:26.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:26.209+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:26.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:26.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:26.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:26.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:26.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:26.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:26.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:26.163+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:27.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:27.065+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:27.065+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:27.065+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:27.065+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:27.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:27.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:27.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:27.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:27.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:27.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:27.204+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:27.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:27.242+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:27.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:27.242+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:27.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:27.242+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:27.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:27.242+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:27.333 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:19:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:19:27.267+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23295 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:19:28.424 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:28.267+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:28.424 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:28.267+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:28.425 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:28.267+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:28.425 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:28.268+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:28.425 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:28.079+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:28.425 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:28.079+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:28.425 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:28.079+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:28.426 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:28.080+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:28.426 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:28.189+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:28.426 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:28.189+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:28.426 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:28.189+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:28.426 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:28.190+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:29.299 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:29.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:29.299 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:29.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:29.300 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:29.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:29.300 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:29.041+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:29.300 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:29.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:29.300 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:29.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:29.300 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:29.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:29.301 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:29.170+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:29.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:29.298+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:29.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:29.298+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:29.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:29.298+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:29.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:29.298+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:30.316 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:30.132+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:30.317 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:30.132+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:30.317 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:30.132+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:30.317 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:30.133+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:30.317 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:29.997+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:30.317 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:29.997+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:30.318 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:29.997+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:30.318 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:29.998+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:30.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:30.315+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:30.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:30.315+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:30.315+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:30.316+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:31.308 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:30.976+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:31.308 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:30.976+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:31.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:30.976+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:31.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:30.977+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:31.309 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:31.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:31.309 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:31.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:31.309 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:31.096+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:31.309 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:31.097+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:31.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:31.306+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:31.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:31.306+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:31.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:31.306+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:31.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:31.307+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:32.271 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:32.014+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:32.272 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:32.014+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:32.272 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:32.014+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:32.272 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:32.015+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:32.272 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:32.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:32.272 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:32.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:32.273 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:32.080+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:32.273 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:32.081+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:32.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:19:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:19:32.270+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23354 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:19:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:32.299+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:32.299+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:32.299+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:32.300+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:33.316 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:33.023+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:33.317 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:33.023+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:33.317 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:33.023+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:33.317 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:33.024+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:33.317 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:33.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:33.318 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:33.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:33.318 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:33.103+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:33.318 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:33.104+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:33.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:33.314+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:33.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:33.314+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:33.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:33.314+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:33.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:33.315+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:34.073+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:34.073+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:34.073+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:34.074+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:34.008+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:34.008+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:34.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:34.008+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:34.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:34.008+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:34.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:34.348+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:34.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:34.348+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:34.348+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:34.349+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:35.301 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:35.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:35.301 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:35.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:35.301 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:35.055+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:35.302 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:35.056+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:35.303 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:34.980+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:35.303 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:34.980+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:35.303 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:34.980+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:35.303 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:34.981+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:35.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:35.299+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:35.299+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:35.299+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:35.300+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:36.288 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:35.945+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:36.288 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:35.945+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:36.289 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:35.945+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:36.289 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:35.946+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:36.289 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:36.057+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:36.289 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:36.057+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:36.289 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:36.057+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:36.289 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:36.057+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:36.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:36.287+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:36.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:36.287+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:36.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:36.287+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:36.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:36.288+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:37.274 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:36.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:37.274 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:36.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:37.274 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:36.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:37.274 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:36.996+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:37.275 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:37.097+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:37.275 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:37.097+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:37.275 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:37.097+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:37.275 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:37.098+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:37.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:19:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:19:37.272+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23412 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:19:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:37.278+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:37.278+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:37.278+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:37.279+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:38.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:38.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:38.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:38.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:38.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:38.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:38.309 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:38.018+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:38.310 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:38.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:38.310 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:38.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:38.310 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:38.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:38.310 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:38.065+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:38.307+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:38.307+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:38.307+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:38.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:38.308+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:39.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:39.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:39.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:39.060+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:39.043+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:39.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:39.043+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:39.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:39.043+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:39.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:39.044+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:39.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:39.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:39.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:39.358+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:40.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:40.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:40.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:40.106+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:40.044+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:40.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:40.044+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:40.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:40.044+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:40.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:40.044+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:40.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:40.382+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:40.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:40.382+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:40.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:40.382+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:40.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:40.383+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:41.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:41.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:41.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:41.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:41.107+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:41.004+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:41.004+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:41.004+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:41.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:41.005+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:41.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:41.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:41.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:41.420+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:42.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:42.144+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:42.144+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:42.144+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:42.144+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:42.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:42.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:42.029+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:42.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:42.030+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:42.332 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:19:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:19:42.274+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23470 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:19:42.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:42.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:42.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:42.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:42.449+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:43.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:43.164+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:43.165+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:43.165+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:43.165+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:43.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:43.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:43.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:43.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:43.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:43.032+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:43.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:43.431+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:43.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:43.431+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:43.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:43.431+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:43.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:43.431+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:44.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:44.193+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:44.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:44.193+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:44.193+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:44.194+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:44.059+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:44.059+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:44.059+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:44.060+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:44.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:44.464+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:44.464+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:44.464+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:44.465+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:45.169+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:45.169+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:45.169+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:45.169+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:45.023+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:45.023+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:45.023+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:45.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:45.024+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:45.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:45.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:45.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:45.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:45.471+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:46.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:46.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:46.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:46.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:46.186+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:46.072+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:46.072+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:46.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:46.072+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:46.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:46.073+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:46.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:46.429+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:46.429+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:46.429+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:46.430+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:47.145 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:47.036+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:47.145 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:47.036+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:47.145 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:47.036+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:47.145 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:47.037+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:47.423 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:19:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:19:47.276+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23529 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:19:47.423 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:47.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:47.423 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:47.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:47.424 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:47.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:47.424 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:47.143+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:47.420+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:47.420+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:47.420+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:47.421+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:48.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:48.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:48.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:48.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:48.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:48.166+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:48.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:48.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:48.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:48.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:48.002+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:48.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:48.447+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:48.447+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:48.447+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:48.447+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:49.211+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:49.211+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:49.211+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:49.211+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:49.039+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:49.039+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:49.039+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:49.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:49.040+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:49.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:49.429+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:49.429+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:49.429+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:49.429+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:50.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:50.201+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:50.201+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:50.201+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:50.202+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:50.012+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:50.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:50.012+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:50.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:50.012+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:50.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:50.013+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:50.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:50.420+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:50.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:50.420+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:50.420+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:50.421+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:51.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:51.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:51.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:51.158+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:50.990+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:51.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:50.990+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:51.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:50.990+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:51.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:50.991+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:51.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:51.464+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:51.464+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:51.464+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:51.465+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:52.198 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:52.026+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:52.199 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:52.026+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:52.199 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:52.026+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:52.199 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:52.027+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:52.450 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:19:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:19:52.279+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23588 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:19:52.451 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:52.197+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:52.451 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:52.197+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:52.451 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:52.197+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:52.451 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:52.198+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:52.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:52.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:52.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:52.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:52.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:52.450+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:53.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:53.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:53.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:53.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:53.156+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:52.996+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:52.996+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:52.996+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:53.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:52.996+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:53.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:53.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:53.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:53.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:53.500+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:54.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:54.146+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:54.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:54.146+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:54.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:54.146+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:54.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:54.147+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:54.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:53.981+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:54.334 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:53.981+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:54.334 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:53.981+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:54.334 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:53.982+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:54.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:54.451+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:54.451+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:54.451+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:54.452+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:55.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:55.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:55.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:55.142+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:55.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:54.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:55.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:54.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:55.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:54.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:55.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:54.943+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:55.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:55.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:55.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:55.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:55.476+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:56.093+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:56.093+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:56.093+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:56.093+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:55.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:55.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:56.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:55.960+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:56.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:55.960+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:56.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:56.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:56.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:56.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:56.479+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:57.283 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:57.133+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:57.284 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:57.133+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:57.284 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:57.133+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:57.284 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:57.134+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:57.284 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:56.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:57.284 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:56.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:57.285 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:56.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:57.285 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:56.973+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:57.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:19:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:19:57.282+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23646 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:19:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:57.452+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:57.452+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:57.452+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:57.453+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:58.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:57.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:57.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:57.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:57.929+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:58.150+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:58.150+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:58.150+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:58.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:58.151+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:58.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:58.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:58.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:58.415+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:58.416+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:19:59.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:59.103+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:59.103+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:59.103+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:19:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:19:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:19:59.104+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:58.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:58.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:58.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:19:59.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:58.900+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:19:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:59.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:59.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:59.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:19:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:19:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:19:59.457+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:00.141 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:59.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:00.141 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:59.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:00.141 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:59.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:00.142 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:19:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:19:59.871+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:00.442 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:00.139+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:00.442 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:00.139+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:00.443 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:00.139+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:00.443 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:00.140+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:00.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:00.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:00.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:00.441+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:00.441+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:01.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:01.137+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:01.137+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:01.137+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:01.138+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:00.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:00.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:00.908+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:00.909+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:01.728 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:01.462+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:01.728 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:01.462+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:01.728 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:01.462+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:01.728 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:01.463+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:02.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:01.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:01.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:01.942+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:01.943+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:02.462 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:20:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:20:02.284+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23705 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:20:02.462 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:02.159+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:02.463 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:02.159+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:02.463 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:02.159+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:02.463 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:02.160+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:02.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:02.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:02.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:02.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:02.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:02.461+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:02.974+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:02.974+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:02.974+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:02.974+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:03.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:03.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:03.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:03.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:03.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:03.128+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:03.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:03.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:03.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:03.455+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:04.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:04.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:04.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:04.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:04.152+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:04.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:04.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:04.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:04.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:04.010+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:04.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:04.010+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:04.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:04.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:04.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:04.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:04.505+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:05.035+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:05.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:05.035+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:05.035+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:05.036+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:05.199+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:05.199+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:05.199+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:05.200+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:05.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:05.531+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:05.531+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:05.531+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:05.532+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:06.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:06.223+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:06.223+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:06.223+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:06.224+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:06.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:06.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:06.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:06.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:06.067+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:06.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:06.067+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:06.817 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:06.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:06.817 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:06.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:06.817 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:06.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:06.818 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:06.557+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:07.571 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:20:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:20:07.286+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23764 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:20:07.572 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:07.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:07.572 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:07.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:07.572 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:07.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:07.572 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:07.195+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:07.573 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:07.084+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:07.573 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:07.084+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:07.573 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:07.084+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:07.573 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:07.085+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:07.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:07.570+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:07.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:07.570+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:07.570+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:07.571+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:08.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:08.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:08.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:08.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:08.149+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:08.123+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:08.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:08.123+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:08.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:08.123+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:08.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:08.124+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:09.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:08.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:09.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:08.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:08.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:08.597+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:09.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:09.088+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:09.088+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:09.088+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:09.089+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:09.191+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:09.191+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:09.191+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:09.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:09.192+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:10.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:09.597+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:10.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:09.597+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:09.597+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:09.597+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:10.570 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:10.195+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:10.570 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:10.195+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:10.570 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:10.195+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:10.570 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:10.196+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:10.570 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:10.130+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:10.571 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:10.130+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:10.571 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:10.130+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:10.571 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:10.130+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:10.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:10.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:10.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:10.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:10.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:10.569+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:11.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:11.154+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:11.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:11.154+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:11.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:11.154+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:11.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:11.154+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:11.235+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:11.235+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:11.235+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:11.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:11.235+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:11.908 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:11.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:11.908 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:11.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:11.908 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:11.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:11.909 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:11.582+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:12.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:20:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:20:12.289+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23823 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:20:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:12.239+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:12.239+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:12.239+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:12.240+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:12.187+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:12.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:12.187+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:12.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:12.187+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:12.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:12.188+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:13.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:12.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:13.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:12.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:13.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:12.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:13.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:12.588+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:13.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:13.285+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:13.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:13.285+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:13.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:13.285+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:13.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:13.286+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:13.153+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:13.153+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:13.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:13.153+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:13.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:13.154+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:14.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:13.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:14.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:13.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:14.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:13.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:14.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:13.626+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:14.496 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:14.258+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:14.497 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:14.258+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:14.497 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:14.258+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:14.497 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:14.259+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:14.497 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:14.158+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:14.497 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:14.158+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:14.498 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:14.158+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:14.498 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:14.159+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:14.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:14.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:14.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:14.634+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:15.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:15.303+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:15.303+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:15.303+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:15.304+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:15.170+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:15.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:15.170+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:15.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:15.170+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:15.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:15.171+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:16.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:15.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:16.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:15.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:16.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:15.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:16.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:15.654+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:16.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:16.287+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:16.287+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:16.287+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:16.288+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:16.142+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:16.142+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:16.142+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:16.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:16.143+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:17.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:16.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:17.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:16.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:17.002 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:16.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:17.002 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:16.659+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:17.293 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:17.164+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:17.294 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:17.164+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:17.294 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:17.164+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:17.294 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:17.164+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:17.569 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:20:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:20:17.291+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23881 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:20:17.569 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:17.332+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:17.570 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:17.332+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:17.570 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:17.332+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:17.570 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:17.333+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:18.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:17.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:18.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:17.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:17.709+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:17.709+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:18.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:18.210+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:18.210+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:18.210+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:18.211+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:18.353+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:18.353+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:18.353+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:18.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:18.354+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:19.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:18.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:18.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:18.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:18.754+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:19.164+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:19.164+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:19.164+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:19.165+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:19.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:19.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:19.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:19.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:19.403+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:20.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:19.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:20.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:19.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:19.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:19.788+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:20.436 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:20.144+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:20.436 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:20.144+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:20.436 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:20.144+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:20.436 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:20.144+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:20.749 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:20.433+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:20.749 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:20.433+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:20.749 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:20.433+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:20.749 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:20.434+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:21.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:20.746+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:21.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:20.746+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:20.746+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:20.747+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:21.410 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:21.138+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:21.410 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:21.138+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:21.410 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:21.138+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:21.410 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:21.138+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:21.736 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:21.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:21.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:21.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:21.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:21.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:21.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:21.408+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:22.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:21.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:21.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:21.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:21.735+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:22.370 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:22.185+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:22.370 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:22.185+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:22.371 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:22.185+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:22.371 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:22.186+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:22.371 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:20:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:20:22.293+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23940 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:20:22.689 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:22.368+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:22.689 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:22.368+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:22.690 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:22.368+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:22.690 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:22.368+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:23.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:22.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:23.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:22.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:22.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:22.711+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:23.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:23.382+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:23.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:23.382+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:23.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:23.382+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:23.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:23.383+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:23.228+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:23.228+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:23.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:23.228+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:23.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:23.228+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:24.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:23.687+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:23.687+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:23.687+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:23.688+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:24.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:24.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:24.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:24.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:24.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:24.273+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:24.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:24.274+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:24.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:24.426+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:24.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:24.426+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:24.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:24.426+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:24.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:24.427+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:25.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:24.672+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:25.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:24.673+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:25.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:24.673+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:25.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:24.673+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:25.469+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:25.469+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:25.469+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:25.469+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:25.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:25.314+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:25.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:25.314+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:25.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:25.314+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:25.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:25.315+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:26.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:25.672+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:25.672+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:25.672+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:25.672+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:26.459+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:26.459+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:26.459+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:26.460+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:26.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:26.267+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:26.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:26.267+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:26.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:26.267+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:26.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:26.268+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:27.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:26.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:26.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:26.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:26.696+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:27.487 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:27.248+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:27.487 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:27.248+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:27.488 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:27.248+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:27.488 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:27.249+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:27.488 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:20:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:20:27.296+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 23998 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:20:27.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:27.486+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:27.486+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:27.486+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:27.486+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:27.676+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:27.676+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:27.676+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:27.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:27.677+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:28.526 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:28.253+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:28.527 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:28.253+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:28.527 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:28.253+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:28.527 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:28.254+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:28.525+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:28.525+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:28.525+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:28.526+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:28.668+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:28.668+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:28.668+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:28.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:28.669+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:29.572 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:29.261+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:29.572 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:29.261+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:29.572 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:29.261+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:29.572 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:29.262+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:29.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:29.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:29.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:29.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:29.571+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:29.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:29.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:29.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:29.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:29.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:29.685+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:30.540 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:30.262+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:30.540 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:30.262+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:30.540 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:30.262+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:30.541 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:30.263+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:30.539+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:30.539+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:30.539+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:30.539+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:30.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:30.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:30.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:30.649+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:31.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:31.514+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:31.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:31.514+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:31.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:31.514+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:31.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:31.515+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:31.312+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:31.312+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:31.312+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:31.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:31.312+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:32.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:31.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:31.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:31.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:31.682+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:32.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:20:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:20:32.299+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 24056 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:20:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:32.502+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:32.502+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:32.502+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:32.502+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:32.346+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:32.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:32.346+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:32.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:32.346+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:32.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:32.346+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:33.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:32.732+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:33.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:32.732+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:32.732+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:32.732+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:33.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:33.382+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:33.382+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:33.382+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:33.383+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:33.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:33.474+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:33.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:33.474+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:33.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:33.474+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:33.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:33.474+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:34.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:33.763+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:33.763+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:33.763+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:33.763+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:34.766 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:34.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:34.766 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:34.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:34.767 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:34.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:34.767 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:34.487+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:34.767 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:34.377+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:34.767 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:34.377+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:34.767 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:34.377+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:34.767 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:34.377+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:35.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:34.764+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:35.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:34.764+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:34.764+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:34.765+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:35.806 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:35.526+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:35.807 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:35.526+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:35.807 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:35.526+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:35.807 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:35.526+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:35.807 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:35.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:35.807 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:35.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:35.807 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:35.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:35.808 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:35.372+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:36.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:35.805+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:35.805+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:35.805+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:35.806+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:36.768 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:36.525+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:36.769 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:36.525+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:36.769 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:36.525+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:36.769 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:36.526+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:36.772 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:36.366+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:36.772 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:36.367+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:36.773 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:36.367+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:36.773 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:36.367+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:37.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:36.767+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:37.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:36.767+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:37.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:36.767+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:37.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:36.768+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:37.567 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:20:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:20:37.302+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 24114 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:20:37.567 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:37.409+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:37.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:37.409+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:37.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:37.409+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:37.568 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:37.410+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:37.565+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:37.565+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:37.565+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:37.565+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:37.803+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:37.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:37.803+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:37.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:37.803+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:37.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:37.803+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:38.557+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:38.557+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:38.557+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:38.558+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:38.396+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:38.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:38.396+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:38.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:38.396+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:38.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:38.397+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:39.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:38.852+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:39.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:38.852+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:39.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:38.852+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:39.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:38.852+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:39.823 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:39.587+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:39.824 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:39.587+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:39.824 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:39.587+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:39.824 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:39.587+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:39.824 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:39.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:39.824 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:39.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:39.825 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:39.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:39.825 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:39.441+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:40.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:39.822+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:39.822+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:39.822+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:39.823+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:40.801 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:40.540+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:40.801 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:40.540+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:40.802 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:40.540+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:40.802 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:40.541+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:40.802 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:40.464+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:40.802 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:40.464+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:40.802 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:40.464+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:40.802 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:40.464+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:41.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:40.799+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:40.799+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:40.799+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:40.800+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:41.829 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:41.557+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:41.829 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:41.557+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:41.829 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:41.557+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:41.829 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:41.557+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:41.829 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:41.508+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:41.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:41.508+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:41.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:41.508+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:41.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:41.509+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:42.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:41.827+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:41.827+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:41.827+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:41.828+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:42.443 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:20:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:20:42.304+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 24173 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:20:42.781 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:42.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:42.782 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:42.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:42.782 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:42.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:42.782 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:42.532+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:42.782 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:42.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:42.782 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:42.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:42.783 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:42.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:42.783 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:42.555+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:43.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:42.780+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:43.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:42.780+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:42.780+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:42.781+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:43.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:43.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:43.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:43.547+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:43.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:43.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:43.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:43.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:43.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:43.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:43.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:43.544+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:43.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:43.789+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:43.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:43.789+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:43.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:43.789+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:43.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:43.790+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:44.541+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:44.541+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:44.541+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:44.542+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:44.541+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:44.541+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:44.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:44.541+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:44.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:44.542+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:44.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:44.769+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:44.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:44.769+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:44.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:44.769+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:44.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:44.769+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:45.813 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:45.501+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:45.813 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:45.501+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:45.813 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:45.501+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:45.813 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:45.502+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:45.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:45.514+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:45.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:45.514+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:45.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:45.514+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:45.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:45.515+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:46.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:45.811+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:46.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:45.811+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:45.811+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:45.812+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:46.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:46.477+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:46.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:46.477+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:46.477+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:46.478+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:46.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:46.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:46.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:46.488+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:47.306 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:46.861+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:47.306 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:46.861+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:47.306 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:46.861+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:47.307 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:46.862+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:47.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:47.472+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:47.472+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:47.472+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:47.472+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:47.455+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:47.455+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:47.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:47.455+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:47.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:47.456+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:47.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:20:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:20:47.307+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 24232 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:20:48.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:47.910+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:48.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:47.910+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:48.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:47.910+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:48.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:47.911+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:48.443+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:48.443+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:48.443+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:48.444+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:48.435+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:48.435+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:48.435+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:48.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:48.436+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:49.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:48.944+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:49.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:48.944+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:49.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:48.944+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:49.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:48.945+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:49.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:49.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:49.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:49.452+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:49.445+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:49.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:49.445+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:49.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:49.445+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:49.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:49.445+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:50.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:49.964+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:50.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:49.964+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:50.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:49.964+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:50.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:49.964+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:50.423+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:50.423+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:50.423+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:50.423+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:50.473+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:50.473+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:50.473+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:50.474+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:51.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:50.944+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:50.944+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:50.944+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:50.944+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:51.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:51.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:51.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:51.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:51.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:51.503+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:51.383+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:51.383+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:51.383+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:51.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:51.383+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:52.311 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:51.915+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:52.311 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:51.915+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:52.311 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:51.915+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:52.311 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:51.916+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:52.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:20:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:20:52.309+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 24291 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:20:52.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:52.379+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:52.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:52.379+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:52.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:52.379+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:52.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:52.380+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:52.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:52.468+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:52.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:52.468+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:52.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:52.468+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:52.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:52.469+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:53.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:52.941+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:53.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:52.942+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:53.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:52.942+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:53.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:52.942+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:53.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:53.475+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:53.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:53.475+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:53.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:53.475+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:53.666 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:53.476+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:53.666 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:53.379+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:53.666 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:53.379+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:53.666 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:53.379+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:53.666 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:53.380+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:54.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:53.950+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:54.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:53.950+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:54.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:53.950+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:54.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:53.950+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:54.414+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:54.414+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:54.414+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:54.414+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:54.513+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:54.513+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:54.513+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:54.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:54.514+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:55.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:54.910+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:55.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:54.910+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:55.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:54.910+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:55.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:54.911+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:55.460+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:55.460+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:55.460+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:55.460+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:55.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:55.504+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:55.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:55.504+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:55.504+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:55.505+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:56.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:55.956+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:56.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:55.956+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:56.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:55.956+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:56.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:55.956+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:56.499+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:56.499+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:56.499+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:56.500+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:56.488+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:56.488+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:56.488+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:56.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:56.489+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:57.313 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:56.939+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:57.313 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:56.939+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:57.313 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:56.939+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:57.314 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:56.939+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:57.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:20:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:20:57.310+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 24349 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:20:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:57.528+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:57.528+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:57.528+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:57.528+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:57.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:57.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:57.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:57.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:57.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:57.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:57.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:57.507+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:58.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:57.965+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:58.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:57.965+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:58.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:57.965+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:58.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:57.966+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:58.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:58.466+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:58.466+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:58.466+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:58.467+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:58.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:58.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:58.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:58.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:58.528+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:59.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:58.974+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:59.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:58.974+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:59.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:58.974+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:20:59.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:58.975+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:20:59.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:59.474+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:59.474+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:59.474+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:20:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:20:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:20:59.474+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:20:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:59.501+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:59.501+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:59.501+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:20:59.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:20:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:20:59.502+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:00.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:59.975+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:00.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:59.975+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:00.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:59.975+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:00.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:20:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:20:59.976+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:00.532+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:00.532+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:00.532+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:00.533+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:00.425+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:00.425+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:00.425+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:00.426+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:01.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:00.995+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:01.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:00.995+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:01.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:00.995+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:01.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:00.996+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:01.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:01.523+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:01.523+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:01.523+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:01.523+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:01.451+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:01.451+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:01.451+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:01.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:01.452+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:02.316 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:01.999+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:02.316 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:01.999+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:02.316 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:01.999+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:02.316 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:01.999+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:02.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:21:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:21:02.313+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 24408 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:21:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:02.510+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:02.510+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:02.510+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:02.510+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:02.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:02.466+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:02.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:02.466+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:02.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:02.466+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:02.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:02.467+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:03.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:02.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:02.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:02.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:02.999+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:03.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:03.508+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:03.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:03.508+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:03.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:03.508+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:03.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:03.509+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:03.462+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:03.462+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:03.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:03.462+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:03.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:03.464+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:04.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:03.973+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:03.973+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:03.973+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:03.974+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:04.418+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:04.418+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:04.418+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:04.418+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:04.532+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:04.532+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:04.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:04.532+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:04.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:04.533+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:05.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:05.024+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:05.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:05.024+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:05.024+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:05.024+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:05.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:05.388+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:05.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:05.388+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:05.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:05.388+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:05.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:05.388+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:05.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:05.544+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:05.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:05.544+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:05.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:05.544+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:05.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:05.545+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:06.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:06.019+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:06.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:06.019+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:06.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:06.019+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:06.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:06.020+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:06.721 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:06.373+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:06.721 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:06.373+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:06.721 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:06.373+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:06.722 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:06.373+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:06.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:06.523+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:06.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:06.523+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:06.723 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:06.523+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:06.723 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:06.524+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:07.318 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:07.023+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:07.318 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:07.023+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:07.318 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:07.023+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:07.319 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:07.024+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:07.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:07.345+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:07.345+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:07.345+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:07.345+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:07.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:07.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:07.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:07.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:07.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:07.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:07.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:07.528+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:07.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:21:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:21:07.316+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 24467 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:21:08.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:08.013+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:08.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:08.013+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:08.013+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:08.014+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:08.769 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:08.509+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:08.770 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:08.509+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:08.770 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:08.509+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:08.770 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:08.509+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:08.771 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:08.342+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:08.771 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:08.342+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:08.771 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:08.342+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:08.771 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:08.342+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:09.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:09.062+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:09.062+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:09.062+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:09.063+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:09.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:09.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:09.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:09.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:09.522+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:09.370+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:09.370+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:09.370+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:09.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:09.371+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:10.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:10.025+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:10.025+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:10.025+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:10.026+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:10.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:10.397+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:10.397+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:10.397+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:10.397+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:10.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:10.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:10.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:10.521+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:11.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:11.006+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:11.006+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:11.006+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:11.006+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:11.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:11.447+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:11.447+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:11.447+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:11.448+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:11.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:11.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:11.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:11.521+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:12.320 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:12.040+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:12.321 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:12.040+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:12.321 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:12.040+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:12.321 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:12.040+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:12.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:12.435+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:12.435+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:12.435+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:12.436+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:12.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:12.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:12.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:12.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:12.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:12.526+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:12.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:21:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:21:12.319+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 24526 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:21:13.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:13.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:13.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:13.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:13.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:13.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:13.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:13.076+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:13.429+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:13.429+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:13.429+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:13.430+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:13.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:13.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:13.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:13.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:13.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:13.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:13.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:13.507+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:14.445 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:14.109+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:14.445 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:14.109+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:14.445 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:14.109+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:14.446 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:14.110+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:14.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:14.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:14.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:14.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:14.440+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:14.515+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:14.515+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:14.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:14.515+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:14.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:14.516+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:15.478 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:15.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:15.478 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:15.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:15.478 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:15.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:15.478 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:15.152+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:15.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:15.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:15.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:15.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:15.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:15.477+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:15.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:15.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:15.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:15.558+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:16.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:16.127+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:16.127+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:16.127+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:16.128+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:16.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:16.475+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:16.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:16.475+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:16.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:16.475+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:16.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:16.476+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:16.518+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:16.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:16.518+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:16.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:16.518+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:16.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:16.519+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:17.466 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:21:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:21:17.321+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 24584 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:21:17.467 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:17.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:17.467 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:17.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:17.467 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:17.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:17.467 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:17.089+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:17.465+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:17.465+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:17.465+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:17.465+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:17.546+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:17.546+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:17.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:17.546+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:17.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:17.547+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:18.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:18.136+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:18.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:18.136+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:18.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:18.136+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:18.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:18.136+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:18.505+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:18.505+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:18.505+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:18.506+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:18.580+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:18.580+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:18.580+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:18.581+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:19.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:19.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:19.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:19.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:19.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:19.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:19.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:19.098+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:19.555+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:19.555+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:19.555+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:19.556+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:19.602+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:19.602+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:19.602+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:19.602+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:20.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:20.062+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:20.062+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:20.062+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:20.062+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:20.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:20.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:20.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:20.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:20.574+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:20.516+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:20.516+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:20.516+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:20.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:20.516+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:21.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:21.038+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:21.038+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:21.038+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:21.039+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:21.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:21.516+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:21.516+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:21.516+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:21.517+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:21.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:21.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:21.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:21.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:21.557+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:22.325 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:22.038+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:22.325 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:22.038+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:22.326 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:22.038+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:22.326 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:22.039+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:22.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:21:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:21:22.324+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 24643 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:21:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:22.534+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:22.534+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:22.534+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:22.534+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:22.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:22.524+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:22.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:22.524+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:22.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:22.524+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:22.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:22.524+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:23.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:23.037+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:23.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:23.037+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:23.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:23.037+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:23.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:23.038+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:23.508+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:23.508+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:23.508+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:23.509+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:23.536+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:23.536+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:23.536+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:23.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:23.536+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:24.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:24.037+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:24.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:24.037+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:24.037+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:24.038+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:24.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:24.462+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:24.462+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:24.462+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:24.463+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:24.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:24.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:24.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:24.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:24.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:24.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:24.545+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:25.153 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:25.014+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:25.153 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:25.014+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:25.153 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:25.014+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:25.153 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:25.015+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:25.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:25.530+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:25.530+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:25.530+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:25.531+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:25.491+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:25.491+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:25.491+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:25.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:25.492+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:26.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:26.057+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:26.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:26.057+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:26.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:26.057+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:26.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:26.058+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:26.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:26.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:26.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:26.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:26.511+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:26.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:26.474+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:26.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:26.474+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:26.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:26.474+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:26.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:26.474+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:27.522 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:21:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:21:27.326+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 24702 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:21:27.523 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:27.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:27.523 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:27.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:27.523 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:27.088+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:27.523 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:27.088+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:27.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:27.521+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:27.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:27.521+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:27.521+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:27.522+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:27.532+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:27.532+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:27.532+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:27.532+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:28.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:28.060+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:28.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:28.060+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:28.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:28.060+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:28.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:28.061+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:28.542+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:28.542+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:28.542+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:28.543+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:28.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:28.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:28.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:28.573+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:29.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:29.018+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:29.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:29.018+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:29.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:29.018+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:29.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:29.019+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:29.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:29.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:29.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:29.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:29.549+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:29.539+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:29.539+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:29.539+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:29.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:29.540+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:30.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:30.000+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:30.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:30.000+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:30.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:30.000+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:30.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:30.001+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:30.575+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:30.575+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:30.575+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:30.576+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:30.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:30.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:30.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:30.528+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:31.297 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:30.954+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:31.297 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:30.954+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:31.297 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:30.954+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:31.297 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:30.954+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:31.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:31.494+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:31.494+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:31.494+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:31.495+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:31.557+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:31.557+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:31.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:31.557+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:31.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:31.557+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:32.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:31.937+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:32.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:31.937+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:32.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:31.937+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:32.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:31.938+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:32.597 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:21:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:21:32.328+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 24761 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:21:32.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:32.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:32.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:32.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:32.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:32.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:32.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:32.479+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:32.939 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:32.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:32.939 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:32.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:32.939 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:32.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:32.939 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:32.597+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:33.278 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:32.937+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:33.278 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:32.937+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:33.278 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:32.937+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:33.278 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:32.938+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:33.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:33.470+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:33.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:33.470+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:33.470+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:33.471+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:33.951 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:33.640+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:33.951 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:33.640+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:33.951 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:33.640+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:33.951 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:33.640+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:34.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:33.948+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:34.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:33.948+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:34.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:33.948+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:34.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:33.949+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:34.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:34.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:34.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:34.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:34.650 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:34.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:34.650 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:34.521+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:34.986 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:34.646+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:34.986 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:34.646+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:34.986 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:34.646+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:34.986 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:34.647+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:35.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:34.983+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:35.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:34.983+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:35.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:34.983+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:35.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:34.984+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:35.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:35.653+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:35.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:35.653+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:35.653+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:35.654+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:35.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:35.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:35.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:35.507+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:36.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:36.033+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:36.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:36.033+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:36.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:36.033+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:36.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:36.034+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:36.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:36.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:36.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:36.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:36.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:36.550+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:36.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:36.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:36.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:36.665+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:37.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:37.000+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:37.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:37.000+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:37.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:37.000+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:37.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:37.001+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:37.676 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:37.559+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:37.676 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:37.559+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:37.676 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:37.559+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:37.676 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:37.560+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:37.677 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:21:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:21:37.330+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 24819 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:21:38.041 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:37.673+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:38.041 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:37.673+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:38.041 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:37.673+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:38.041 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:37.674+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:38.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:38.038+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:38.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:38.038+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:38.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:38.038+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:38.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:38.039+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:38.664 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:38.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:38.664 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:38.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:38.664 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:38.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:38.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:38.568+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:39.028 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:38.661+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:39.028 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:38.661+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:39.028 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:38.661+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:39.029 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:38.662+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:39.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:39.025+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:39.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:39.025+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:39.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:39.025+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:39.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:39.026+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:40.000 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:39.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:40.000 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:39.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:40.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:39.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:40.001 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:39.618+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:40.001 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:39.591+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:40.001 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:39.591+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:40.001 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:39.591+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:40.001 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:39.592+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:40.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:39.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:40.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:39.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:40.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:39.998+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:40.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:39.998+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:40.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:40.621+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:40.621+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:40.621+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:40.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:40.622+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:40.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:40.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:40.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:40.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:40.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:40.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:40.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:40.606+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:41.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:40.974+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:41.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:40.974+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:41.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:40.974+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:41.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:40.975+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:41.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:41.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:41.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:41.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:41.569+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:41.654+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:41.654+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:41.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:41.654+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:41.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:41.655+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:42.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:42.014+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:42.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:42.014+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:42.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:42.014+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:42.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:42.015+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:42.644 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:21:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:21:42.333+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 24878 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:21:42.644 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:42.565+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:42.644 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:42.565+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:42.645 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:42.565+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:42.645 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:42.565+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:43.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:42.642+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:43.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:42.642+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:43.058 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:42.642+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:43.058 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:42.643+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:43.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:43.056+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:43.056+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:43.056+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:43.057+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:43.517+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:43.517+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:43.517+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:43.518+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:43.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:43.686+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:43.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:43.686+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:43.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:43.686+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:43.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:43.687+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:44.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:44.010+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:44.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:44.010+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:44.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:44.010+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:44.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:44.010+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:44.512+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:44.512+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:44.512+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:44.512+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:44.657+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:44.657+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:44.657+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:44.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:44.658+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:45.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:45.035+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:45.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:45.035+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:45.035+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:45.035+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:45.673 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:45.490+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:45.674 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:45.490+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:45.674 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:45.490+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:45.674 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:45.491+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:46.005 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:45.672+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:46.006 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:45.672+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:46.006 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:45.672+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:46.006 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:45.673+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:46.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:46.004+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:46.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:46.004+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:46.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:46.004+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:46.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:46.004+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:46.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:46.452+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:46.452+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:46.452+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:46.453+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:46.661+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:46.661+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:46.661+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:46.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:46.663+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:47.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:47.028+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:47.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:47.028+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:47.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:47.028+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:47.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:47.029+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:47.692 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:21:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:21:47.336+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 24937 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:21:47.693 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:47.468+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:47.693 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:47.468+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:47.693 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:47.468+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:47.693 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:47.469+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:48.022 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:47.691+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:48.022 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:47.691+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:48.023 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:47.691+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:48.023 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:47.692+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:48.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:48.020+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:48.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:48.020+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:48.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:48.020+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:48.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:48.021+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:48.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:48.722+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:48.722+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:48.722+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:48.723+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:48.501+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:48.501+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:48.501+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:48.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:48.502+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:49.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:49.049+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:49.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:49.049+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:49.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:49.049+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:49.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:49.050+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:49.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:49.718+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:49.718+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:49.718+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:49.719+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:49.503+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:49.503+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:49.503+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:49.504+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:50.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:50.055+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:50.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:50.055+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:50.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:50.055+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:50.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:50.056+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:50.512+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:50.512+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:50.513+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:50.513+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:50.760+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:50.760+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:50.760+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:50.761+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:51.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:51.033+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:51.033+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:51.033+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:51.033+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:51.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:51.789+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:51.789+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:51.789+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:51.790+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:51.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:51.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:51.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:51.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:51.552+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:52.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:52.070+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:52.070+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:52.070+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:52.070+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:52.743 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:21:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:21:52.338+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 24996 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:21:52.744 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:52.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:52.744 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:52.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:52.744 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:52.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:52.744 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:52.571+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:53.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:52.742+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:52.742+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:52.742+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:52.742+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:53.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:53.120+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:53.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:53.120+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:53.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:53.120+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:53.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:53.121+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:54.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:53.718+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:53.718+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:53.718+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:53.719+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:53.607+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:53.607+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:54.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:53.607+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:54.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:53.608+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:54.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:54.093+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:54.093+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:54.093+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:54.094+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:55.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:54.637+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:55.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:54.637+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:55.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:54.637+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:55.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:54.638+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:55.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:54.742+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:55.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:54.742+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:55.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:54.742+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:55.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:54.742+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:55.361 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:55.102+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:55.361 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:55.102+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:55.361 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:55.102+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:55.361 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:55.102+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:56.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:55.642+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:55.642+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:55.642+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:55.642+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:55.726+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:55.726+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:55.726+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:56.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:55.727+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:56.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:56.139+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:56.139+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:56.139+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:56.140+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:57.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:56.615+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:56.615+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:56.615+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:56.615+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:56.742+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:56.742+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:56.742+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:57.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:56.743+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:57.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:21:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:21:57.341+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 25054 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:21:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:57.163+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:57.163+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:57.163+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:57.164+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:58.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:57.582+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:57.582+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:57.582+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:57.582+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:57.752+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:57.752+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:57.752+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:58.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:57.753+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:58.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:58.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:58.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:58.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:58.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:58.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:58.152+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:21:59.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:58.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:58.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:58.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:21:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:58.606+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:58.717+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:58.717+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:59.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:58.717+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:21:59.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:58.718+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:21:59.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:59.134+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:59.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:59.134+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:59.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:59.134+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:21:59.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:21:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:21:59.135+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:00.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:59.717+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:00.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:59.717+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:00.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:59.717+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:00.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:21:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:21:59.718+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:00.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:59.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:00.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:59.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:00.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:59.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:00.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:21:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:21:59.626+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:00.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:00.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:00.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:00.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:00.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:00.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:00.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:00.098+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:00.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:00.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:00.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:00.605+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:00.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:00.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:00.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:01.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:00.767+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:01.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:01.111+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:01.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:01.111+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:01.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:01.111+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:01.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:01.112+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:02.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:01.753+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:01.753+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:01.753+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:01.753+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:01.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:01.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:01.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:02.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:01.626+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:02.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:22:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:22:02.344+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 25113 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:22:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:02.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:02.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:02.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:02.120+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:02.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:02.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:02.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:02.611+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:02.802+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:02.802+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:02.802+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:02.803+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:03.577 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:03.149+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:03.577 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:03.149+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:03.577 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:03.149+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:03.577 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:03.149+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:03.828 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:03.575+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:03.828 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:03.575+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:03.829 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:03.575+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:03.829 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:03.576+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:04.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:03.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:03.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:03.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:03.827+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:04.563 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:04.101+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:04.564 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:04.101+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:04.564 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:04.101+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:04.564 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:04.102+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:04.562+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:04.562+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:04.562+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:04.563+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:05.088 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:04.829+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:05.088 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:04.829+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:05.089 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:04.829+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:05.089 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:04.830+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:05.575 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:05.111+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:05.575 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:05.111+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:05.575 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:05.111+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:05.575 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:05.112+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:05.573+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:05.573+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:05.573+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:05.574+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:05.818+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:05.818+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:05.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:05.818+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:05.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:05.819+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:06.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:06.108+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:06.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:06.108+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:06.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:06.108+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:06.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:06.109+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:07.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:06.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:06.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:06.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:06.597+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:07.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:06.821+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:07.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:06.821+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:07.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:06.821+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:07.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:06.822+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:07.568 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:22:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:22:07.346+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 25172 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:22:07.568 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:07.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:07.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:07.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:07.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:07.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:07.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:07.119+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:07.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:07.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:07.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:07.567+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:08.120 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:07.837+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:08.120 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:07.837+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:08.120 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:07.837+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:08.120 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:07.838+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:08.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:08.118+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:08.118+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:08.118+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:08.119+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:08.915 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:08.594+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:08.915 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:08.594+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:08.915 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:08.594+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:08.915 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:08.595+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:08.916 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:08.811+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:08.916 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:08.811+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:08.916 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:08.811+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:08.916 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:08.812+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:09.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:09.087+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:09.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:09.087+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:09.087+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:09.088+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:10.077 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:09.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:10.077 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:09.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:10.077 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:09.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:10.078 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:09.596+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:10.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:09.800+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:10.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:09.800+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:10.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:09.800+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:10.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:09.800+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:10.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:10.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:10.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:10.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:10.076+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:10.845 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:10.593+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:10.845 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:10.593+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:10.845 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:10.593+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:10.845 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:10.594+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:11.234 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:10.843+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:11.235 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:10.843+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:11.235 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:10.843+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:11.235 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:10.844+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:11.235 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:11.039+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:11.235 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:11.039+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:11.235 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:11.039+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:11.236 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:11.039+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:12.046 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:11.794+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:12.046 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:11.794+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:12.046 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:11.794+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:12.046 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:11.795+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:12.047 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:11.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:12.047 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:11.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:12.047 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:11.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:12.047 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:11.626+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:12.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:12.043+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:12.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:12.043+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:12.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:12.043+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:12.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:12.044+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:12.670 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:22:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:22:12.348+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 25231 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:22:13.054 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:12.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:13.054 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:12.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:13.054 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:12.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:13.054 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:12.668+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:13.054 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:12.785+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:13.055 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:12.785+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:13.055 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:12.785+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:13.055 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:12.786+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:13.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:13.051+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:13.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:13.051+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:13.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:13.051+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:13.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:13.052+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:13.996 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:13.628+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:13.996 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:13.628+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:13.997 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:13.628+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:13.997 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:13.629+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:13.997 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:13.769+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:13.997 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:13.769+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:13.997 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:13.769+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:13.998 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:13.770+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:14.304 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:14.023+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:14.305 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:14.023+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:14.305 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:14.023+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:14.305 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:14.024+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:15.057 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:14.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:15.057 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:14.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:15.057 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:14.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:15.057 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:14.591+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:15.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:14.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:15.058 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:14.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:15.058 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:14.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:15.058 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:14.784+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:15.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:15.054+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:15.054+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:15.054+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:15.055+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:15.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:15.554+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:15.554+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:15.554+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:15.555+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:15.740+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:15.740+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:15.740+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:15.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:15.740+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:16.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:16.068+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:16.068+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:16.068+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:16.069+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:17.079 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:16.597+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:17.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:16.597+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:17.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:16.597+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:17.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:16.597+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:17.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:16.755+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:17.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:16.755+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:17.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:16.755+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:16.755+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:17.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:17.078+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:17.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:17.078+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:17.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:17.078+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:17.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:17.078+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:17.736 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:22:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:22:17.351+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 25290 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:22:17.736 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:17.559+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:17.736 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:17.559+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:17.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:17.559+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:17.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:17.560+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:18.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:17.734+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:18.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:17.734+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:18.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:17.734+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:18.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:17.735+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:18.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:18.112+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:18.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:18.112+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:18.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:18.112+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:18.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:18.112+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:19.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:18.704+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:19.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:18.704+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:18.704+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:18.705+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:18.585+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:18.585+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:18.585+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:19.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:18.586+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:19.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:19.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:19.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:19.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:19.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:19.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:19.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:19.120+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:20.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:19.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:19.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:19.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:19.754+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:19.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:19.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:20.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:19.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:20.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:19.628+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:20.449 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:20.156+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:20.449 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:20.156+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:20.449 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:20.156+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:20.450 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:20.156+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:21.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:20.734+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:20.734+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:20.734+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:20.734+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:20.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:21.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:20.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:21.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:20.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:21.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:20.597+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:21.551 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:21.126+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:21.551 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:21.126+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:21.552 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:21.126+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:21.552 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:21.126+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:21.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:21.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:21.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:21.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:21.550+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:21.784+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:21.784+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:21.784+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:21.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:21.785+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:22.575 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:22.104+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:22.575 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:22.104+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:22.576 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:22.104+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:22.576 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:22.105+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:22.576 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:22:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:22:22.354+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 25347 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:22:22.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:22.778+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:22.778+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:22.778+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:22.778+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:22.573+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:22.573+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:22.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:22.573+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:22.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:22.574+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:23.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:23.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:23.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:23.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:23.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:23.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:23.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:23.075+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:23.811 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:23.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:23.811 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:23.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:23.812 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:23.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:23.812 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:23.548+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:24.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:23.810+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:24.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:23.810+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:24.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:23.810+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:24.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:23.810+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:24.048+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:24.048+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:24.048+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:24.048+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:24.764 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:24.510+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:24.764 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:24.510+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:24.765 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:24.510+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:24.765 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:24.510+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:25.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:24.762+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:25.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:24.762+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:25.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:24.762+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:25.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:24.763+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:25.463 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:25.084+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:25.463 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:25.084+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:25.464 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:25.084+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:25.464 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:25.085+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:25.748 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:25.462+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:25.748 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:25.462+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:25.748 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:25.462+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:25.748 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:25.462+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:26.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:25.746+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:26.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:25.746+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:26.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:25.746+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:26.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:25.747+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:26.446 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:26.134+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:26.446 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:26.134+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:26.446 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:26.134+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:26.446 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:26.135+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:26.704 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:26.444+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:26.704 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:26.444+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:26.705 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:26.444+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:26.705 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:26.445+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:27.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:26.703+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:26.703+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:26.703+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:26.703+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:27.359 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:27.094+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:27.360 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:27.094+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:27.360 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:27.094+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:27.360 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:27.094+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:27.655 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:22:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:22:27.356+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 25407 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:22:27.656 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:27.419+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:27.656 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:27.419+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:27.656 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:27.419+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:27.656 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:27.420+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:28.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:27.654+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:28.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:27.654+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:27.654+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:27.655+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:28.466 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:28.084+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:28.466 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:28.084+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:28.467 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:28.084+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:28.467 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:28.085+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:28.465+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:28.465+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:28.465+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:28.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:28.466+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:28.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:28.635+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:28.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:28.635+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:28.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:28.635+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:28.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:28.636+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:29.498 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:29.102+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:29.498 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:29.102+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:29.498 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:29.102+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:29.499 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:29.102+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:29.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:29.496+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:29.496+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:29.496+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:29.497+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:29.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:29.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:29.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:29.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:29.605+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:30.524 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:30.138+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:30.524 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:30.138+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:30.524 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:30.138+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:30.524 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:30.138+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:30.522+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:30.522+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:30.522+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:30.523+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:30.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:30.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:30.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:30.582+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:31.524 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:31.126+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:31.525 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:31.126+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:31.525 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:31.126+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:31.525 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:31.127+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:31.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:31.523+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:31.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:31.523+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:31.523+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:31.524+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:31.571+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:31.571+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:31.571+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:31.572+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:32.525 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:22:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:22:32.359+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 25465 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:22:32.525 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:32.146+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:32.525 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:32.146+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:32.526 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:32.146+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:32.526 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:32.147+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:32.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:32.523+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:32.523+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:32.523+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:32.524+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:32.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:32.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:32.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:32.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:32.556+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:33.499 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:33.115+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:33.499 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:33.115+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:33.500 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:33.115+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:33.500 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:33.116+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:33.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:33.498+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:33.498+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:33.498+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:33.499+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:33.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:33.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:33.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:33.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:33.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:33.563+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:34.548 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:34.158+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:34.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:34.158+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:34.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:34.158+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:34.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:34.159+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:34.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:34.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:34.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:34.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:34.548+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:34.571+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:34.571+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:34.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:34.571+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:34.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:34.571+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:35.567 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:35.129+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:35.568 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:35.129+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:35.568 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:35.129+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:35.568 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:35.130+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:35.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:35.575+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:35.575+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:35.575+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:35.576+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:35.566+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:35.566+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:35.566+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:35.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:35.567+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:36.560 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:36.174+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:36.561 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:36.175+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:36.561 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:36.175+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:36.561 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:36.175+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:36.559+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:36.559+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:36.559+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:36.560+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:36.592+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:36.592+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:36.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:36.592+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:36.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:36.592+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:37.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:22:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:22:37.361+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 25524 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:22:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:37.129+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:37.129+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:37.129+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:37.130+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:37.589+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:37.589+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:37.589+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:37.590+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:37.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:37.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:37.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:38.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:37.617+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:38.567 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:38.160+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:38.568 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:38.160+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:38.568 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:38.160+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:38.568 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:38.160+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:38.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:38.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:38.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:38.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:38.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:38.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:38.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:38.567+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:38.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:38.632+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:38.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:38.632+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:38.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:38.632+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:38.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:38.633+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:39.486 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:39.131+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:39.486 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:39.131+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:39.486 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:39.131+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:39.487 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:39.131+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:39.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:39.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:39.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:39.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:39.592+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:39.636+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:39.636+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:39.636+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:39.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:39.637+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:40.553 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:40.099+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:40.553 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:40.099+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:40.554 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:40.099+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:40.554 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:40.099+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:40.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:40.595+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:40.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:40.595+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:40.595+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:40.596+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:40.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:40.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:40.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:40.552+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:41.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:41.072+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:41.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:41.072+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:41.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:41.072+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:41.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:41.073+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:41.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:41.598+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:41.598+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:41.598+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:41.599+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:41.535+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:41.535+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:41.535+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:41.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:41.536+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:42.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:42.048+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:42.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:42.048+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:42.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:42.048+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:42.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:42.048+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:42.639 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:22:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:22:42.364+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 25582 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:22:42.640 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:42.526+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:42.640 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:42.526+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:42.640 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:42.526+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:42.640 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:42.526+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:43.032 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:42.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:43.032 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:42.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:43.032 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:42.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:43.032 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:42.639+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:43.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:43.030+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:43.030+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:43.030+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:43.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:43.031+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:43.743 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T04:22:43.740+0000 7f60e50fc640 0 monclient(hunting): authenticate timed out after 300 2024-09-10T04:22:43.743 INFO:teuthology.orchestra.run.smithi112.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-09-10T04:22:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:43.541+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:43.541+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:43.541+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:43.541+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:43.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:43.590+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:43.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:43.590+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:43.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:43.590+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:43.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:43.591+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:44.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:44.069+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:44.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:44.069+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:44.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:44.069+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:44.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:44.070+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:44.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T04:22:44.545 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -- ceph osd blacklist ls 2024-09-10T04:22:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:44.535+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:44.535+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:44.535+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:44.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:44.536+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:44.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:44.628+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:44.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:44.628+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:44.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:44.628+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:44.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:44.629+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:44.838 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/mon.smithi112/config 2024-09-10T04:22:45.236 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:45.042+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:45.236 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:45.042+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:45.237 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:45.042+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:45.237 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:45.043+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:45.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:45.505+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:45.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:45.505+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:45.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:45.505+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:45.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:45.506+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:45.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:45.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:45.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:45.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:45.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:45.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:45.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:45.631+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:46.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:46.025+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:46.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:46.025+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:46.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:46.025+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:46.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:46.026+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:46.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:46.610+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:46.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:46.610+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:46.610+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:46.611+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:46.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:46.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:46.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:46.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:46.549+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:47.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:47.030+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:47.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:47.030+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:47.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:47.030+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:47.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:47.030+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:47.743 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:22:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:22:47.366+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 25642 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:22:47.744 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:47.517+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:47.744 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:47.517+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:47.744 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:47.517+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:47.744 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:47.518+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:47.744 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:47.576+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:47.744 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:47.576+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:47.745 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:47.576+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:47.745 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:47.576+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:48.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:48.046+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:48.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:48.046+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:48.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:48.046+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:48.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:48.046+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:48.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:48.548+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:48.548+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:48.548+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:48.549+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:48.485+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:48.485+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:48.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:48.485+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:48.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:48.485+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:49.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:49.052+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:49.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:49.052+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:49.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:49.052+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:49.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:49.053+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:49.533+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:49.533+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:49.533+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:49.533+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:49.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:49.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:49.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:49.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:49.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:49.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:49.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:49.569+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:50.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:50.065+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:50.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:50.065+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:50.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:50.065+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:50.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:50.066+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:50.495+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:50.495+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:50.495+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:50.496+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:50.526+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:50.526+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:50.526+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:50.527+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:51.522 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:51.092+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:51.522 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:51.092+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:51.522 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:51.092+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:51.523 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:51.093+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:51.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:51.543+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:51.543+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:51.543+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:51.544+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:51.519+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:51.519+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:51.519+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:51.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:51.520+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:52.540 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:22:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:22:52.368+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 25700 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:22:52.540 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:52.141+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:52.540 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:52.141+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:52.541 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:52.141+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:52.541 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:52.141+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:52.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:52.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:52.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:52.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:52.546+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:52.537+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:52.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:52.537+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:52.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:52.537+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:52.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:52.538+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:53.536 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:53.158+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:53.536 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:53.158+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:53.537 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:53.158+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:53.537 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:53.159+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:53.534+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:53.534+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:53.534+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:53.534+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:53.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:53.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:53.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:53.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:53.550+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:54.542 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:54.183+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:54.542 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:54.183+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:54.543 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:54.183+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:54.543 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:54.184+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:54.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:54.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:54.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:54.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:54.594+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:54.541+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:54.541+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:54.541+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:54.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:54.541+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:55.570 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:55.195+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:55.570 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:55.195+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:55.571 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:55.195+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:55.571 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:55.196+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:55.569+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:55.892 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:55.569+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:55.892 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:55.569+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:55.892 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:55.569+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:55.893 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:55.579+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:55.893 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:55.579+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:55.893 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:55.579+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:55.893 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:55.580+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:56.540 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:56.171+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:56.540 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:56.171+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:56.540 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:56.171+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:56.541 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:56.171+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:56.538+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:56.538+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:56.538+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:56.539+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:56.612+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:56.612+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:56.612+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:56.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:56.612+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:57.525 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:22:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:22:57.372+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 25758 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:22:57.525 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:57.213+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:57.525 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:57.213+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:57.525 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:57.213+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:57.525 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:57.214+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:57.825 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:57.523+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:57.825 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:57.523+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:57.826 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:57.523+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:57.826 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:57.524+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:57.826 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:57.570+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:57.826 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:57.570+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:57.826 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:57.570+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:57.826 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:57.571+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:58.572 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:58.247+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:58.572 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:58.247+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:58.572 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:58.247+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:58.573 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:58.248+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:58.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:58.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:58.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:58.571+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:58.600+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:58.600+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:58.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:58.600+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:58.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:58.601+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:59.532 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:59.216+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:59.532 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:59.216+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:59.533 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:59.216+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:22:59.533 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:22:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:22:59.216+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:22:59.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:59.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:59.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:59.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:22:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:22:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:22:59.531+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:22:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:59.640+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:59.640+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:59.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:59.640+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:22:59.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:22:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:22:59.641+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:00.557 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:00.173+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:00.557 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:00.173+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:00.557 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:00.173+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:00.557 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:00.173+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:00.555+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:00.555+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:00.555+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:00.556+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:00.642+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:00.642+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:00.642+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:00.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:00.643+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:01.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:01.136+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:01.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:01.136+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:01.570 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:01.136+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:01.570 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:01.136+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:01.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:01.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:01.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:01.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:01.568+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:01.596+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:01.596+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:01.596+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:01.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:01.597+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:02.531 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:23:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:23:02.374+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 25816 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:23:02.532 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:02.157+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:02.532 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:02.157+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:02.532 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:02.157+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:02.532 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:02.158+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:02.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:02.530+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:02.530+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:02.530+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:02.531+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:02.579+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:02.579+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:02.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:02.579+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:02.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:02.579+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:03.516 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:03.181+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:03.516 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:03.181+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:03.516 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:03.181+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:03.517 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:03.182+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:03.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:03.514+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:03.514+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:03.514+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:03.515+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:03.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:03.614+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:03.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:03.614+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:03.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:03.614+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:03.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:03.615+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:04.548 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:04.214+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:04.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:04.214+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:04.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:04.214+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:04.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:04.215+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:04.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:04.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:04.547+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:04.548+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:04.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:04.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:04.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:04.639+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:05.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:05.175+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:05.175+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:05.175+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:05.176+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:05.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:05.625+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:05.625+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:05.625+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:05.626+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:05.540+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:05.540+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:05.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:05.540+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:05.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:05.540+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:06.529 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:06.209+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:06.529 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:06.209+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:06.529 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:06.209+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:06.530 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:06.210+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:06.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:06.580+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:06.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:06.580+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:06.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:06.580+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:06.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:06.581+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:06.641+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:06.641+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:06.641+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:06.642+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:07.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:23:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:23:07.377+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 25876 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:23:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:07.245+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:07.245+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:07.245+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:07.245+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:07.594+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:07.594+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:07.594+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:07.595+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:07.675+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:07.675+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:07.675+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:08.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:07.675+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:08.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:08.273+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:08.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:08.273+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:08.273+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:08.274+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:08.909 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:08.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:08.909 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:08.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:08.909 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:08.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:08.909 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:08.587+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:08.909 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:08.695+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:08.910 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:08.695+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:08.910 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:08.695+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:08.910 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:08.696+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:09.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:09.291+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:09.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:09.291+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:09.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:09.291+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:09.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:09.292+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:10.053 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:09.580+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:10.053 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:09.580+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:10.053 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:09.580+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:10.053 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:09.580+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:10.053 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:09.727+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:10.054 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:09.727+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:10.054 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:09.727+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:10.054 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:09.727+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:10.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:10.295+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:10.295+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:10.295+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:10.296+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:10.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:10.540+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:10.540+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:10.540+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:10.540+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:10.741+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:10.741+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:10.741+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:10.742+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:11.537 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:11.262+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:11.537 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:11.262+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:11.538 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:11.262+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:11.538 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:11.263+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:11.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:11.536+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:11.536+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:11.536+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:11.536+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:11.772+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:11.772+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:11.772+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:11.773+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:12.570 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:23:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:23:12.379+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 25933 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:23:12.570 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:12.302+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:12.570 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:12.302+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:12.571 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:12.302+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:12.571 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:12.302+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:12.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:12.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:12.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:12.569+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:12.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:12.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:12.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:12.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:12.783+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:13.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:13.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:13.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:13.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:13.260+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:14.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:13.778+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:13.778+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:13.778+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:13.779+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:13.587+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:13.587+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:14.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:13.587+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:14.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:13.587+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:14.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:14.290+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:14.290+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:14.290+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:14.291+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:15.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:14.588+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:14.588+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:14.588+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:14.589+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:15.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:14.745+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:15.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:14.745+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:15.084 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:14.745+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:15.084 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:14.746+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:15.555 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:15.262+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:15.556 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:15.263+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:15.556 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:15.263+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:15.556 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:15.263+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:15.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:15.554+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:15.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:15.554+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:15.554+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:15.555+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:15.722+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:15.722+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:15.722+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:15.723+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:16.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:16.301+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:16.301+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:16.301+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:16.302+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:17.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:16.583+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:16.583+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:16.583+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:16.583+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:16.682+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:16.682+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:17.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:16.682+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:17.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:16.683+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:17.546 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:23:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:23:17.382+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 25992 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:23:17.547 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:17.264+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:17.547 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:17.264+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:17.547 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:17.264+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:17.547 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:17.265+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:17.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:17.701+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:17.701+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:17.701+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:17.702+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:17.545+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:17.545+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:17.545+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:17.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:17.546+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:18.568 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:18.247+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:18.568 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:18.247+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:18.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:18.247+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:18.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:18.247+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:18.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:18.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:18.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:18.568+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:18.694+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:18.694+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:18.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:18.694+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:18.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:18.695+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:19.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:19.241+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:19.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:19.241+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:19.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:19.241+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:19.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:19.242+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:19.592+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:19.592+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:19.592+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:19.593+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:19.655+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:19.655+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:19.655+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:19.656+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:20.568 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:20.264+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:20.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:20.264+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:20.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:20.264+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:20.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:20.264+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:20.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:20.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:20.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:20.568+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:20.649+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:20.649+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:20.649+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:20.649+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:21.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:21.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:21.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:21.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:21.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:21.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:21.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:21.313+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:21.558+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:21.558+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:21.558+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:21.558+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:21.889 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:21.632+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:21.889 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:21.632+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:21.889 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:21.632+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:21.890 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:21.632+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:22.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:23:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:23:22.385+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 26050 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:23:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:22.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:22.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:22.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:22.548+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:22.305+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:22.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:22.305+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:22.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:22.305+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:22.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:22.306+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:23.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:22.674+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:22.674+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:22.674+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:22.675+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:23.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:23.308+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:23.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:23.308+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:23.308+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:23.309+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:24.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:23.629+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:24.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:23.629+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:24.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:23.629+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:24.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:23.629+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:23.584+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:23.584+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:23.584+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:24.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:23.584+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:24.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:24.303+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:24.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:24.303+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:24.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:24.303+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:24.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:24.303+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:24.960 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:24.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:24.961 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:24.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:24.961 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:24.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:24.961 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:24.627+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:24.961 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:24.643+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:24.962 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:24.643+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:24.962 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:24.643+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:24.962 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:24.643+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:25.343+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:25.343+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:25.343+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:25.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:25.343+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:26.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:25.687+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:26.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:25.687+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:26.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:25.687+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:26.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:25.688+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:25.632+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:25.632+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:26.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:25.632+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:26.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:25.633+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:26.626 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:26.334+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:26.626 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:26.334+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:26.626 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:26.334+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:26.627 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:26.335+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:26.623+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:26.623+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:26.623+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:26.624+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:26.647+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:27.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:26.647+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:27.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:26.647+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:27.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:26.648+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:27.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:23:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:23:27.386+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 26110 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:23:27.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:27.576+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:27.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:27.576+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:27.576+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:27.326+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:27.326+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:27.326+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:27.327+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:27.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:27.577+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:27.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:27.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:27.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:27.617+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:28.573 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:28.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:28.573 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:28.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:28.574 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:28.570+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:28.574 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:28.318+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:28.574 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:28.318+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:28.574 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:28.318+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:28.574 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:28.319+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:28.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:28.571+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:28.585+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:28.585+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:28.585+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:28.586+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:29.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:29.269+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:29.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:29.269+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:29.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:29.269+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:29.570 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:29.270+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:29.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:29.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:29.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:29.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:29.574+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:29.633+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:29.633+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:29.633+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:29.633+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:30.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:30.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:30.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:30.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:30.314+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:30.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:30.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:30.605+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:30.605+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:30.680+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:30.680+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:31.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:30.680+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:31.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:30.680+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:31.563 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:31.282+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:31.563 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:31.282+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:31.563 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:31.282+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:31.563 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:31.283+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:31.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:31.560+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:31.560+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:31.560+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:31.561+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:31.730+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:31.730+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:31.730+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:31.731+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:32.545 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:23:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:23:32.390+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 26168 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:23:32.545 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:32.278+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:32.545 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:32.278+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:32.546 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:32.278+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:32.546 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:32.278+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:32.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:32.543+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:32.543+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:32.543+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:32.544+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:32.696+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:32.696+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:32.696+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:32.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:32.697+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:33.516 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:33.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:33.516 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:33.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:33.516 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:33.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:33.516 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:33.260+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:33.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:33.725+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:33.725+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:33.725+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:33.725+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:33.511+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:33.511+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:33.511+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:33.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:33.512+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:34.507 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:34.242+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:34.507 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:34.242+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:34.507 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:34.242+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:34.508 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:34.242+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:34.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:34.505+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:34.505+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:34.505+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:34.506+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:34.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:34.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:34.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:34.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:34.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:34.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:34.755+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:35.533 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:35.249+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:35.534 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:35.249+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:35.534 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:35.249+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:35.534 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:35.250+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:35.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:35.532+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:35.532+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:35.532+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:35.533+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:35.750+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:35.750+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:35.750+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:35.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:35.751+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:36.529 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:36.202+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:36.529 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:36.202+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:36.530 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:36.202+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:36.530 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:36.203+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:36.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:36.707+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:36.707+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:36.707+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:36.708+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:36.528+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:36.528+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:36.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:36.528+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:36.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:36.529+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:37.494 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:37.185+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:37.495 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:37.185+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:37.495 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:37.185+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:37.495 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:37.185+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:37.495 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:23:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:23:37.392+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 26227 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:23:37.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:37.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:37.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:37.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:37.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:37.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:37.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:37.494+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:37.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:37.667+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:37.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:37.667+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:37.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:37.667+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:37.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:37.668+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:38.473 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:38.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:38.474 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:38.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:38.474 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:38.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:38.474 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:38.222+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:38.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:38.471+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:38.471+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:38.471+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:38.472+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:38.632+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:38.632+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:38.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:38.632+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:38.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:38.632+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:39.478 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:39.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:39.478 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:39.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:39.478 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:39.222+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:39.478 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:39.222+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:39.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:39.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:39.476+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:39.477+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:39.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:39.625+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:39.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:39.625+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:39.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:39.625+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:39.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:39.626+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:40.554 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:40.509+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:40.554 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:40.509+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:40.554 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:40.509+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:40.555 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:40.510+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:40.555 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:40.269+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:40.555 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:40.269+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:40.555 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:40.269+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:40.555 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:40.270+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:40.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:40.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:40.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:40.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:40.639+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:41.556 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:41.255+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:41.557 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:41.255+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:41.557 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:41.255+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:41.557 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:41.256+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:41.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:41.555+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:41.555+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:41.555+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:41.556+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:41.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:41.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:41.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:41.620+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:42.545 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:23:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:23:42.395+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 26285 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:23:42.546 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:42.267+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:42.546 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:42.267+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:42.546 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:42.267+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:42.546 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:42.267+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:42.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:42.597+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:42.597+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:42.597+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:42.598+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:42.544+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:42.544+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:42.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:42.544+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:42.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:42.544+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:43.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:43.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:43.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:43.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:43.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:43.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:43.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:43.311+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:44.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:43.588+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:43.588+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:43.588+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:43.589+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:44.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:43.626+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:44.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:43.626+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:44.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:43.626+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:44.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:43.626+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:44.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:44.281+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:44.281+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:44.281+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:44.282+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:45.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:44.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:44.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:44.618+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:44.618+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:45.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:44.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:45.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:44.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:45.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:44.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:45.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:44.604+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:45.562 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:45.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:45.563 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:45.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:45.563 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:45.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:45.563 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:45.297+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:45.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:45.579+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:45.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:45.579+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:45.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:45.579+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:45.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:45.580+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:45.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:45.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:45.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:45.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:45.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:45.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:45.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:45.562+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:46.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:46.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:46.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:46.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:46.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:46.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:46.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:46.296+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:46.526+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:46.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:46.526+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:46.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:46.526+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:46.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:46.526+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:47.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:46.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:46.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:46.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:46.614+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:47.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:23:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:23:47.398+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 26344 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:23:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:47.304+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:47.304+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:47.304+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:47.304+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:47.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:47.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:47.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:47.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:47.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:47.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:47.490+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:48.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:47.657+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:47.657+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:47.657+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:47.658+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:48.660 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:48.336+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:48.660 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:48.336+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:48.660 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:48.336+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:48.660 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:48.336+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:48.661 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:48.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:48.661 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:48.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:48.661 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:48.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:48.661 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:48.526+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:48.658+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:48.658+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:48.658+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:49.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:48.659+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:49.700 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:49.478+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:49.700 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:49.478+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:49.700 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:49.478+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:49.700 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:49.479+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:49.701 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:49.355+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:49.701 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:49.355+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:49.701 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:49.355+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:49.701 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:49.355+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:50.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:49.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:49.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:49.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:49.699+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:50.726 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:50.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:50.726 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:50.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:50.726 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:50.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:50.727 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:50.522+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:50.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:50.347+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:50.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:50.347+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:50.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:50.347+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:50.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:50.348+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:51.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:50.740+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:50.740+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:50.740+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:50.741+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:51.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:51.365+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:51.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:51.365+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:51.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:51.365+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:51.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:51.365+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:51.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:51.570+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:51.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:51.570+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:51.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:51.570+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:51.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:51.571+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:52.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:51.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:52.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:51.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:51.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:51.732+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:52.768 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:23:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:23:52.400+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 26402 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:23:52.768 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:52.342+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:52.768 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:52.342+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:52.768 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:52.342+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:52.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:52.342+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:52.769 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:52.534+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:52.769 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:52.534+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:52.769 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:52.534+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:52.769 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:52.534+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:52.770 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:52.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:52.770 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:52.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:52.770 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:52.766+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:52.770 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:52.767+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:53.679 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:53.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:53.680 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:53.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:53.680 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:53.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:53.680 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:53.330+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:53.680 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:53.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:53.680 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:53.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:53.680 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:53.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:53.681 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:53.503+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:54.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:53.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:54.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:53.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:53.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:53.732+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:54.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:54.512+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:54.512+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:54.512+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:54.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:54.512+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:54.299+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:54.299+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:54.299+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:54.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:54.300+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:55.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:54.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:55.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:54.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:55.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:54.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:55.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:54.684+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:55.647 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:55.507+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:55.648 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:55.507+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:55.648 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:55.507+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:55.648 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:55.508+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:55.648 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:55.338+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:55.648 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:55.338+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:55.649 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:55.338+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:55.649 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:55.338+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:56.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:55.643+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:56.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:55.643+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:55.643+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:55.644+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:56.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:56.290+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:56.290+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:56.290+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:56.290+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:56.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:56.498+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:56.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:56.498+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:56.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:56.498+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:56.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:56.499+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:57.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:56.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:56.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:56.613+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:56.614+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:57.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:23:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:23:57.403+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 26461 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:23:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:57.565+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:57.565+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:57.565+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:57.565+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:57.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:57.320+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:57.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:57.320+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:57.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:57.320+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:57.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:57.321+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:57.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:57.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:57.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:57.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:57.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:57.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:57.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:57.529+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:58.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:58.369+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:58.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:58.369+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:58.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:58.369+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:58.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:58.370+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:58.754 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:58.579+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:58.754 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:58.579+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:58.754 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:58.579+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:58.754 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:58.580+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:58.755 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:58.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:58.755 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:58.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:58.755 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:58.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:58.755 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:58.556+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:59.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:59.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:59.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:59.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:59.779 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:59.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:23:59.779 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:23:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:23:59.405+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:23:59.779 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:59.579+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:59.779 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:59.579+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:59.779 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:59.579+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:23:59.779 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:23:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:23:59.580+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:23:59.780 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:59.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:59.780 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:59.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:59.780 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:59.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:23:59.780 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:23:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:23:59.587+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:00.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:00.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:00.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:00.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:00.573+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:00.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:00.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:00.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:00.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:00.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:00.384+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:00.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:00.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:00.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:00.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:00.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:00.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:00.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:00.567+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:01.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:01.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:01.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:01.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:01.526+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:01.582+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:01.582+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:01.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:01.582+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:01.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:01.583+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:01.340+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:01.340+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:01.340+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:01.341+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:02.569 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:24:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:24:02.405+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 26519 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:24:02.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:02.314+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:02.570 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:02.314+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:02.570 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:02.314+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:02.570 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:02.315+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:02.570 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:02.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:02.570 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:02.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:02.570 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:02.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:02.571 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:02.525+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:02.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:02.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:02.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:02.567+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:02.568+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:03.537 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:03.534+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:03.537 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:03.534+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:03.538 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:03.534+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:03.538 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:03.535+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:03.538 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:03.284+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:03.538 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:03.284+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:03.538 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:03.284+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:03.538 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:03.285+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:03.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:03.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:03.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:03.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:03.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:03.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:03.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:03.575+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:04.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:04.302+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:04.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:04.302+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:04.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:04.302+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:04.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:04.303+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:04.896 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:04.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:04.897 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:04.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:04.897 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:04.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:04.897 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:04.617+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:04.897 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:04.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:04.897 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:04.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:04.898 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:04.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:04.898 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:04.579+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:05.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:05.326+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:05.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:05.326+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:05.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:05.326+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:05.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:05.326+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:05.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:05.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:05.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:05.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:05.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:05.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:05.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:05.550+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:05.967 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:05.587+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:05.967 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:05.587+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:05.967 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:05.587+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:05.967 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:05.588+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:06.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:06.298+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:06.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:06.298+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:06.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:06.298+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:06.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:06.299+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:06.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:06.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:06.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:06.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:06.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:06.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:06.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:06.516+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:07.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:06.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:06.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:06.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:06.616+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:07.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:07.514+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:07.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:07.514+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:07.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:07.514+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:07.650 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:07.514+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:07.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:07.342+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:07.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:07.342+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:07.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:07.342+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:07.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:07.343+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:07.651 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:24:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:24:07.407+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 26579 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:24:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:07.646+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:07.646+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:07.646+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:07.647+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:08.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:08.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:08.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:08.313+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:08.531+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:08.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:08.531+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:08.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:08.531+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:08.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:08.531+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:09.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:08.608+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:08.608+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:08.608+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:08.608+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:09.528 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:09.273+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:09.529 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:09.273+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:09.529 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:09.273+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:09.529 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:09.273+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:09.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:09.630+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:09.630+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:09.630+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:09.631+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:09.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:09.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:09.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:09.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:09.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:09.527+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:10.507 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:10.233+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:10.507 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:10.233+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:10.507 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:10.233+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:10.508 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:10.234+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:10.647+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:10.647+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:10.647+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:10.648+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:10.505+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:10.505+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:10.505+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:10.506+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:11.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:11.259+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:11.259+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:11.259+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:11.260+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:11.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:11.515+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:11.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:11.515+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:11.515+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:11.515+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:11.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:11.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:11.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:11.698+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:12.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:24:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:24:12.411+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 26637 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:24:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:12.500+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:12.500+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:12.500+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:12.501+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:12.276+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:12.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:12.276+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:12.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:12.276+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:12.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:12.277+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:13.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:12.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:12.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:12.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:12.733+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:13.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:13.299+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:13.299+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:13.299+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:13.300+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:13.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:13.495+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:13.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:13.495+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:13.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:13.495+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:13.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:13.496+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:14.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:13.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:13.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:13.721+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:13.721+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:14.544 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:14.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:14.545 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:14.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:14.545 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:14.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:14.545 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:14.331+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:14.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:14.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:14.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:14.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:14.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:14.544+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:14.680+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:14.680+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:14.680+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:14.681+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:15.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:15.512+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:15.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:15.512+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:15.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:15.512+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:15.650 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:15.513+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:15.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:15.346+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:15.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:15.346+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:15.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:15.346+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:15.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:15.347+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:16.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:15.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:15.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:15.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:15.680+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:16.684 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:16.394+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:16.685 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:16.394+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:16.685 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:16.394+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:16.685 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:16.394+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:16.685 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:16.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:16.685 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:16.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:16.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:16.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:16.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:16.517+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:17.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:16.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:16.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:16.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:16.684+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:17.552 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:24:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:24:17.413+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 26696 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:24:17.552 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:17.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:17.552 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:17.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:17.553 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:17.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:17.553 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:17.418+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:17.676+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:17.676+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:17.676+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:17.677+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:17.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:17.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:17.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:17.551+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:18.668 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:18.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:18.668 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:18.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:18.668 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:18.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:18.669 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:18.405+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:18.669 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:18.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:18.669 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:18.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:18.669 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:18.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:18.669 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:18.550+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:18.666+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:18.666+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:18.666+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:18.667+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:19.635+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:19.635+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:19.635+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:19.636+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:19.546+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:19.546+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:19.546+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:19.547+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:19.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:19.401+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:19.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:19.401+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:19.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:19.401+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:19.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:19.402+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:20.671 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:20.383+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:20.671 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:20.383+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:20.671 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:20.383+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:20.672 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:20.383+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:20.672 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:20.546+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:20.672 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:20.546+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:20.672 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:20.546+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:20.672 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:20.547+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:21.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:20.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:21.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:20.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:20.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:20.670+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:21.513 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:21.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:21.513 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:21.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:21.514 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:21.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:21.514 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:21.376+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:21.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:21.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:21.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:21.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:21.512+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:21.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:21.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:21.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:21.621+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:22.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:24:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:24:22.416+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 26754 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:24:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:22.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:22.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:22.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:22.587+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:22.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:22.473+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:22.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:22.473+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:22.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:22.473+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:22.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:22.473+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:22.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:22.371+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:22.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:22.371+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:22.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:22.371+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:22.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:22.372+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:23.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:23.455+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:23.455+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:23.455+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:23.455+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:23.329+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:23.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:23.329+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:23.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:23.329+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:23.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:23.330+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:24.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:23.635+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:23.635+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:23.635+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:23.635+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:24.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:24.482+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:24.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:24.482+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:24.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:24.482+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:24.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:24.483+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:24.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:24.324+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:24.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:24.324+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:24.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:24.324+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:24.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:24.325+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:25.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:24.594+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:24.594+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:24.594+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:24.595+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:25.601+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:25.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:25.601+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:25.601+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:25.602+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:25.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:25.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:25.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:25.491+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:25.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:25.372+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:25.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:25.372+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:25.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:25.372+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:25.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:25.373+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:26.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:26.363+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:26.363+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:26.363+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:26.364+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:26.540+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:26.540+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:26.540+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:26.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:26.540+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:27.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:26.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:27.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:26.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:26.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:26.627+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:27.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:24:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:24:27.419+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 26814 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:24:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:27.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:27.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:27.357+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:27.357+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:27.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:27.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:27.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:27.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:27.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:27.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:27.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:27.592+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:27.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:27.517+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:27.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:27.517+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:27.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:27.517+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:27.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:27.517+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:28.552+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:28.553+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:28.553+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:28.553+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:28.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:28.577+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:28.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:28.577+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:28.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:28.577+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:28.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:28.578+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:28.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:28.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:28.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:28.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:28.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:28.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:28.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:28.378+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:29.732 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:29.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:29.732 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:29.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:29.732 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:29.551+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:29.733 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:29.552+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:29.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:29.597+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:29.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:29.597+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:29.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:29.597+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:29.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:29.598+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:29.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:29.412+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:29.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:29.412+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:29.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:29.412+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:29.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:29.412+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:30.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:30.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:30.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:30.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:30.621+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:30.543+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:30.543+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:30.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:30.543+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:30.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:30.543+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:30.394+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:30.394+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:30.394+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:30.394+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:31.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:31.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:31.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:31.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:31.586+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:31.386+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:31.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:31.386+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:31.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:31.386+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:31.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:31.386+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:31.611+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:31.611+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:31.611+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:31.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:31.612+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:32.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:24:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:24:32.421+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 26872 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:24:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:32.607+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:32.607+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:32.607+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:32.608+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:32.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:32.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:32.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:32.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:32.568+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:32.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:32.569+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:32.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:32.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:32.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:32.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:32.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:32.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:32.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:32.418+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:33.683 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:33.598+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:33.683 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:33.598+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:33.683 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:33.598+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:33.683 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:33.598+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:33.683 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:33.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:33.684 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:33.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:33.684 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:33.417+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:33.684 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:33.418+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:33.684 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:33.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:33.684 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:33.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:33.684 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:33.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:33.685 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:33.618+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:34.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:34.449+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:34.449+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:34.449+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:34.450+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:34.661+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:34.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:34.661+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:34.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:34.661+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:34.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:34.662+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:34.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:34.629+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:34.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:34.629+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:34.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:34.629+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:34.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:34.629+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:35.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:35.597+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:35.597+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:35.597+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:35.598+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:35.453+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:35.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:35.453+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:35.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:35.453+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:35.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:35.454+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:35.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:35.654+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:35.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:35.654+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:35.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:35.654+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:35.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:35.654+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:36.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:36.641+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:36.641+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:36.641+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:36.642+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:36.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:36.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:36.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:36.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:36.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:36.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:36.418+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:36.651+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:36.651+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:36.651+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:36.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:36.652+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:37.688 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:37.409+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:37.689 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:37.409+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:37.689 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:37.409+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:37.689 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:37.409+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:37.689 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:37.639+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:37.690 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:37.639+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:37.690 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:37.639+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:37.690 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:37.640+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:37.690 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:24:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:24:37.424+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 26931 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:24:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:37.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:37.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:37.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:37.688+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:38.706 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:38.597+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:38.706 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:38.597+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:38.706 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:38.597+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:38.706 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:38.598+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:38.707 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:38.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:38.707 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:38.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:38.707 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:38.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:38.707 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:38.411+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:39.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:38.704+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:38.704+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:38.704+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:38.705+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:39.664 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:39.558+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:39.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:39.558+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:39.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:39.558+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:39.665 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:39.559+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:39.665 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:39.390+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:39.666 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:39.390+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:39.666 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:39.390+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:39.666 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:39.390+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:40.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:39.663+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:40.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:39.663+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:39.663+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:39.663+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:40.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:40.553+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:40.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:40.553+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:40.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:40.553+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:40.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:40.554+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:40.683 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:40.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:40.683 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:40.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:40.683 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:40.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:40.683 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:40.419+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:41.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:40.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:41.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:40.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:40.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:40.680+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:41.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:41.540+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:41.540+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:41.540+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:41.541+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:41.633+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:41.633+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:41.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:41.633+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:41.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:41.633+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:41.405+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:41.405+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:41.405+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:41.406+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:42.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:24:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:24:42.426+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 26989 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:24:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:42.584+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:42.584+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:42.584+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:13.571154+0000 (oldest deadline 2024-09-10T04:14:33.571154+0000) 2024-09-10T04:24:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:42.585+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:42.409+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:42.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:42.409+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:42.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:42.409+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:42.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:42.409+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:42.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:42.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:42.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:42.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:42.582+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:43.683 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:43.592+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:43.683 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:43.383+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:43.683 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:43.383+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:43.683 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:43.383+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:43.684 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:43.383+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:43.684 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:43.594+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:43.684 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:43.594+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:43.684 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:43.594+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:43.684 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:43.595+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:44.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:44.588+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:44.423+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:44.423+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:44.423+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:44.424+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:44.622+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:44.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:44.622+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:44.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:44.622+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:44.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:44.623+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:45.610 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:45.431+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:45.610 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:45.431+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:45.610 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:45.431+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:45.611 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:45.431+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:46.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:45.607+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:46.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:45.617+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:46.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:45.617+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:46.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:45.617+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:46.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:45.617+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:46.635 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:46.607+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:46.636 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:46.471+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:46.636 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:46.471+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:46.636 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:46.471+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:46.636 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:46.471+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:47.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:46.634+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:46.634+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:46.634+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:46.635+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:47.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:24:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:24:47.429+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 27049 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:24:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:47.628+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:47.431+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:47.431+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:47.431+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:47.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:47.432+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:47.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:47.602+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:47.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:47.602+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:47.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:47.602+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:47.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:47.603+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:48.591+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:48.576+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:48.576+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:48.576+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:48.576+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:48.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:48.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:48.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:48.417+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:49.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:49.638+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:49.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:49.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:49.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:49.578+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:49.401+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:49.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:49.401+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:49.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:49.401+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:49.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:49.401+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:50.651+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:50.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:50.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:50.407+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:50.407+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:50.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:50.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:50.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:50.562+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:51.647 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:51.392+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:51.647 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:51.392+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:51.647 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:51.392+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:18.438452+0000 (oldest deadline 2024-09-10T04:14:38.438452+0000) 2024-09-10T04:24:51.647 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:51.393+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:51.647 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:51.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:51.648 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:51.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:51.648 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:51.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:51.648 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:51.551+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:52.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:51.645+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:52.648 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:52.370+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:52.648 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:24:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:24:52.434+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 27107 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:24:52.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:52.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:52.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:52.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:52.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:52.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:52.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:52.555+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:53.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:52.646+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:53.662 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:53.388+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:53.698 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:53.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:53.698 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:53.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:53.698 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:53.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:53.699 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:53.573+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:54.048 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:53.660+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:54.703 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:54.413+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:54.703 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:54.565+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:54.703 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:54.565+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:54.703 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:54.565+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:14:19.850037+0000 (oldest deadline 2024-09-10T04:14:39.850037+0000) 2024-09-10T04:24:54.703 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:54.566+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:55.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:54.701+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:55.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:55.437+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:55.664+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:55.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:55.613+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:56.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:56.411+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:56.660+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:56.568+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:57.693 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:57.418+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:57.694 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:24:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:24:57.437+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 27166 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:24:57.694 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:57.525+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:58.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:57.692+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:58.728 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:58.536+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:58.728 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:58.448+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:59.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:58.726+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:24:59.736 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:24:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:24:59.411+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:24:59.737 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:24:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:24:59.525+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:00.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:24:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:24:59.735+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:00.707 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:00.438+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:00.708 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:00.504+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:00.705+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:01.729 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:01.498+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:01.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:01.396+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:01.740+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:02.765 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:02.411+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:02.765 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:25:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:25:02.439+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 27224 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:25:02.766 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:02.541+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:02.763+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:03.685 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:03.382+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:03.685 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:03.553+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:04.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:03.724+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:04.700 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:04.515+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:04.700 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:04.390+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:05.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:04.698+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:05.722 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:05.392+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:05.723 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:05.503+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:06.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:05.721+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:06.700 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:06.420+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:06.700 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:06.546+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:07.063 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:06.698+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:07.715 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:07.417+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:07.716 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:07.500+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:07.716 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:25:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:25:07.442+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 27284 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:25:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:07.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:07.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:07.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:07.714+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:08.670 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:08.405+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:08.671 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:08.466+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:08.669+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:08.669+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:08.669+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:08.670+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:09.682 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:09.428+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:09.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:09.429+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:09.680+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:09.680+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:09.680+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:09.681+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:10.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:10.456+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:10.656+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:10.656+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:10.656+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:10.656+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:10.426+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:11.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:11.432+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:11.644+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:11.644+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:11.644+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:11.645+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:11.451+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:12.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:12.474+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:12.474+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:12.474+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:12.475+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:12.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:12.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:12.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:12.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:12.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:12.639+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:12.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:12.461+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:12.832 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:25:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:25:12.445+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 27341 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:25:13.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:13.519+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:13.519+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:13.519+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:13.520+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:13.600+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:13.600+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:13.600+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:13.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:13.601+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:13.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:13.482+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:14.570+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:14.570+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:14.570+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:14.570+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:14.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:14.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:14.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:14.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:14.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:14.587+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:14.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:14.506+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:15.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:15.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:15.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:15.579+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:15.622+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:15.622+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:15.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:15.622+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:15.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:15.623+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:15.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:15.526+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:16.572+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:16.572+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:16.572+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:16.573+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:16.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:16.651+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:16.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:16.651+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:16.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:16.651+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:16.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:16.652+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:16.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:16.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:16.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:16.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:16.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:16.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:16.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:16.556+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:17.544 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:25:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:25:17.447+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 27400 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:25:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:17.641+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:17.641+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:17.641+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:17.642+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:17.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:17.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:17.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:17.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:17.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:17.550+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:17.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:17.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:17.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:17.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:17.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:17.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:17.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:17.543+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:18.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:18.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:18.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:18.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:18.518+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:18.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:18.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:18.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:18.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:18.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:18.627+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:18.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:18.542+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:18.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:18.542+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:18.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:18.542+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:18.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:18.543+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:19.675+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:19.675+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:19.675+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:19.676+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:19.469+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:19.469+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:19.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:19.469+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:19.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:19.470+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:19.576+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:19.576+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:19.576+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:19.577+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:20.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:20.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:20.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:20.511+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:20.642+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:20.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:20.642+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:20.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:20.642+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:20.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:20.643+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:20.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:20.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:20.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:20.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:20.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:20.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:20.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:20.555+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:21.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:21.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:21.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:21.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:21.523+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:21.675+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:21.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:21.675+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:21.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:21.675+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:21.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:21.676+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:21.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:21.507+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:21.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:21.507+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:21.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:21.507+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:21.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:21.508+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:22.721 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:22.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:22.721 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:22.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:22.722 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:22.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:22.722 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:22.503+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:22.722 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:25:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:25:22.450+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 27458 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:25:22.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:22.496+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:22.723 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:22.496+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:22.723 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:22.496+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:22.723 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:22.496+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:22.719+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:22.719+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:22.719+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:22.719+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:23.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:23.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:23.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:23.684+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:23.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:23.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:23.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:23.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:23.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:23.471+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:23.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:23.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:23.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:23.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:23.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:23.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:23.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:23.544+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:24.776 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:24.666+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:24.776 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:24.666+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:24.776 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:24.666+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:24.776 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:24.666+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:24.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:24.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:24.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:24.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:24.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:24.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:24.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:24.546+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:24.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:24.491+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:24.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:24.491+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:24.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:24.491+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:24.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:24.492+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:25.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:25.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:25.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:25.524+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:25.678+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:25.678+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:25.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:25.678+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:25.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:25.678+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:25.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:25.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:25.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:25.533+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:26.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:26.669+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:26.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:26.669+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:26.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:26.669+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:26.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:26.670+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:26.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:26.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:26.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:26.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:26.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:26.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:26.500+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:26.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:26.486+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:26.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:26.486+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:26.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:26.486+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:26.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:26.486+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:27.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:27.634+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:27.634+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:27.634+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:27.634+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:27.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:27.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:27.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:27.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:27.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:27.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:27.462+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:27.832 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:25:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:25:27.454+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 27518 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:25:27.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:27.532+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:27.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:27.532+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:27.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:27.532+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:27.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:27.533+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:28.681 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:28.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:28.681 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:28.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:28.681 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:28.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:28.681 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:28.422+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:28.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:28.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:28.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:28.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:28.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:28.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:28.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:28.517+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:29.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:28.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:28.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:28.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:28.680+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:29.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:29.672+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:29.672+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:29.672+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:29.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:29.673+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:29.428+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:29.428+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:29.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:29.428+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:29.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:29.429+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:29.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:29.547+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:29.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:29.547+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:29.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:29.547+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:29.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:29.548+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:30.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:30.653+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:30.653+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:30.653+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:30.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:30.654+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:30.469+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:30.469+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:30.469+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:30.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:30.470+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:30.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:30.580+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:30.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:30.580+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:30.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:30.580+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:30.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:30.581+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:31.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:31.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:31.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:31.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:31.650 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:31.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:31.650 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:31.555+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:31.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:31.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:31.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:31.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:31.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:31.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:31.650 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:31.479+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:32.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:31.647+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:31.647+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:31.647+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:31.648+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:32.488+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:32.488+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:32.488+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:32.489+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:32.632+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:32.632+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:32.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:32.632+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:32.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:32.633+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:32.832 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:25:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:25:32.457+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 27575 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:25:32.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:32.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:32.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:32.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:32.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:32.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:32.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:32.594+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:33.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:33.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:33.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:33.460+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:33.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:33.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:33.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:33.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:33.617+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:33.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:33.617+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:33.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:33.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:33.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:33.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:33.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:33.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:33.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:33.590+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:34.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:34.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:34.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:34.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:34.417+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:34.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:34.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:34.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:34.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:34.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:34.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:34.587+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:34.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:34.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:34.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:34.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:34.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:34.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:34.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:34.579+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:35.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:35.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:35.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:35.368+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:35.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:35.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:35.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:35.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:35.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:35.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:35.611+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:35.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:35.544+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:35.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:35.544+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:35.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:35.544+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:35.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:35.545+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:36.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:36.651+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:36.651+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:36.651+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:36.651+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:36.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:36.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:36.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:36.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:36.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:36.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:36.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:36.404+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:36.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:36.563+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:36.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:36.563+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:36.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:36.563+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:36.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:36.564+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:37.517 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:37.443+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:37.518 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:37.443+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:37.518 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:37.443+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:37.518 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:37.443+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:37.518 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:25:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:25:37.460+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 27633 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:25:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:37.622+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:37.622+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:37.622+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:37.623+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:37.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:37.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:37.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:37.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:37.517+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:38.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:38.656+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:38.656+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:38.656+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:38.657+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:38.560+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:38.560+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:38.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:38.560+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:38.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:38.561+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:38.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:38.462+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:38.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:38.462+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:38.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:38.462+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:38.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:38.463+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:39.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:39.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:39.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:39.510+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:39.658+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:39.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:39.658+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:39.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:39.658+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:39.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:39.658+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:39.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:39.519+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:39.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:39.519+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:39.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:39.519+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:39.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:39.520+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:40.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:40.567+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:40.567+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:40.567+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:40.568+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:40.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:40.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:40.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:40.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:40.498+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:40.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:40.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:40.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:40.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:40.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:40.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:40.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:40.691+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:41.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:41.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:41.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:41.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:41.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:41.711+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:41.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:41.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:41.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:41.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:41.569+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:41.514+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:41.514+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:41.514+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:41.515+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:42.749 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:25:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:25:42.462+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 27690 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:25:42.749 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:42.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:42.750 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:42.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:42.750 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:42.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:42.750 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:42.555+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:42.750 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:42.547+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:42.750 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:42.547+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:42.750 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:42.547+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:42.750 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:42.548+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:43.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:42.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:42.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:42.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:42.748+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:43.687 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:43.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:43.688 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:43.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:43.688 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:43.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:43.688 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:43.586+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:43.688 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:43.499+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:43.688 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:43.499+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:43.689 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:43.499+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:43.689 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:43.500+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:44.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:43.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:43.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:43.710+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:43.711+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:44.757 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:44.505+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:44.757 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:44.505+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:44.757 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:44.505+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:44.758 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:44.505+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:44.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:44.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:44.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:44.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:44.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:44.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:44.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:44.632+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:45.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:44.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:44.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:44.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:44.756+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:45.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:45.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:45.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:45.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:45.591+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:45.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:45.715+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:45.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:45.715+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:45.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:45.715+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:45.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:45.717+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:45.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:45.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:45.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:45.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:45.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:45.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:45.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:45.525+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:46.716 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:46.606+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:46.716 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:46.606+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:46.717 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:46.606+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:46.717 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:46.606+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:46.717 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:46.508+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:46.717 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:46.508+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:46.717 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:46.508+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:46.718 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:46.509+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:46.714+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:46.714+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:46.714+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:46.716+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:47.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:47.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:47.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:47.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:47.703+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:47.621+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:47.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:47.621+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:47.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:47.621+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:47.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:47.622+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:47.832 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:25:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:25:47.465+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 27750 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:25:47.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:47.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:47.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:47.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:47.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:47.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:47.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:47.491+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:48.725 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:48.472+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:48.726 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:48.472+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:48.726 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:48.472+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:48.726 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:48.473+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:48.726 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:48.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:48.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:48.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:48.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:48.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:48.727 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:48.584+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:49.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:48.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:48.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:48.724+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:48.724+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:49.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:49.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:49.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:49.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:49.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:49.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:49.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:49.450+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:49.722 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:49.565+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:49.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:49.565+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:49.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:49.565+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:49.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:49.565+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:50.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:49.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:50.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:49.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:49.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:49.721+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:50.695 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:50.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:50.695 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:50.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:50.695 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:50.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:50.696 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:50.616+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:50.696 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:50.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:50.696 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:50.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:50.697 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:50.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:50.697 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:50.417+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:51.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:50.693+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:50.693+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:50.693+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:50.694+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:51.718 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:51.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:51.719 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:51.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:51.719 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:51.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:51.719 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:51.582+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:51.719 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:51.432+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:51.719 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:51.432+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:51.720 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:51.432+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:51.720 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:51.433+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:52.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:51.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:51.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:51.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:51.718+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:52.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:52.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:52.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:52.697+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:52.474+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:52.474+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:52.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:52.474+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:52.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:52.475+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:52.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:52.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:52.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:52.592+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:52.833 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:25:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:25:52.468+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 27808 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:25:53.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:53.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:53.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:53.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:53.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:53.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:53.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:53.579+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:53.730 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:53.457+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:53.730 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:53.457+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:53.730 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:53.457+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:53.730 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:53.457+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:54.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:53.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:53.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:53.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:53.728+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:54.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:54.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:54.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:54.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:54.600+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:54.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:54.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:54.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:54.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:54.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:54.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:54.730+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:54.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:54.492+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:54.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:54.492+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:54.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:54.492+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:54.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:54.493+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:55.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:55.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:55.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:55.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:55.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:55.638+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:55.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:55.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:55.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:55.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:55.691+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:55.500+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:55.500+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:55.500+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:55.500+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:56.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:56.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:56.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:56.688+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:56.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:56.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:56.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:56.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:56.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:56.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:56.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:56.678+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:56.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:56.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:56.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:56.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:56.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:56.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:56.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:56.551+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:57.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:25:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:25:57.470+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 27867 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:25:57.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:57.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:57.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:57.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:57.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:57.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:57.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:57.687+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:57.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:57.701+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:57.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:57.701+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:57.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:57.701+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:57.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:57.701+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:57.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:57.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:57.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:57.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:57.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:57.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:57.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:57.550+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:58.697+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:58.697+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:58.697+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:58.698+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:58.508+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:58.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:58.508+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:58.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:58.508+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:58.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:58.509+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:58.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:58.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:58.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:58.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:58.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:58.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:58.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:58.658+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:59.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:59.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:59.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:25:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:25:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:25:59.693+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:25:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:59.739+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:59.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:59.739+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:59.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:59.739+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:25:59.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:25:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:25:59.739+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:25:59.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:59.539+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:59.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:59.539+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:59.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:59.539+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:25:59.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:25:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:25:59.540+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:00.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:00.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:00.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:00.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:00.711+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:00.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:00.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:00.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:00.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:00.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:00.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:00.727+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:00.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:00.493+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:00.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:00.493+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:00.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:00.493+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:00.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:00.494+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:01.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:01.727+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:01.727+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:01.727+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:01.727+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:01.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:01.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:01.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:01.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:01.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:01.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:01.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:01.728+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:01.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:01.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:01.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:01.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:01.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:01.524+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:01.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:01.524+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:02.766 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:26:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:26:02.474+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 27925 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:26:02.767 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:02.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:02.767 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:02.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:02.767 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:02.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:02.767 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:02.550+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:02.765+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:02.765+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:02.765+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:02.766+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:02.768+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:02.768+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:02.768+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:03.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:02.768+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:04.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:03.727+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:03.727+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:03.727+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:03.728+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:03.586+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:04.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:03.586+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:04.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:03.586+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:04.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:03.587+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:04.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:03.716+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:04.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:03.716+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:04.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:03.716+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:04.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:03.717+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:04.897 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:04.707+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:04.898 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:04.707+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:04.898 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:04.707+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:04.898 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:04.708+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:04.898 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:04.687+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:04.898 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:04.687+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:04.898 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:04.687+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:04.899 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:04.687+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:04.899 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:04.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:04.899 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:04.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:04.899 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:04.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:04.899 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:04.605+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:06.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:05.750+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:05.750+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:05.750+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:05.751+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:05.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:05.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:06.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:05.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:06.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:05.669+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:06.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:05.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:06.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:05.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:06.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:05.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:06.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:05.631+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:07.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:06.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:06.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:06.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:06.719+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:07.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:06.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:07.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:06.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:07.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:06.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:07.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:06.668+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:07.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:06.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:07.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:06.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:07.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:06.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:07.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:06.661+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:07.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:07.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:07.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:07.721+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:07.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:07.663+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:07.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:07.663+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:07.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:07.663+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:07.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:07.664+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:07.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:07.675+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:07.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:07.675+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:07.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:07.675+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:07.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:07.676+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:07.833 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:26:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:26:07.477+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 27984 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:26:09.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:08.731+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:08.731+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:08.731+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:08.732+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:09.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:08.699+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:09.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:08.699+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:09.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:08.699+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:09.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:08.699+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:08.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:08.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:08.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:08.632+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:09.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:09.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:09.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:09.742+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:10.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:09.664+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:10.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:09.664+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:10.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:09.664+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:10.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:09.664+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:10.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:09.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:10.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:09.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:10.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:09.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:10.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:09.596+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:10.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:10.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:10.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:10.697+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:10.705+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:10.706+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:10.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:10.706+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:10.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:10.706+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:10.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:10.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:10.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:10.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:10.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:10.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:10.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:10.555+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:12.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:11.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:12.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:11.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:11.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:11.581+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:11.699+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:11.699+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:11.699+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:12.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:11.699+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:12.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:11.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:12.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:11.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:12.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:11.742+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:12.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:11.743+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:12.790 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:26:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:26:12.480+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28042 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:26:12.791 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:12.713+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:12.791 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:12.713+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:12.791 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:12.713+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:12.791 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:12.714+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:12.791 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:12.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:12.792 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:12.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:12.792 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:12.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:12.792 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:12.583+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:13.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:12.789+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:12.789+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:12.789+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:12.790+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:13.684 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:13.576+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:13.684 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:13.576+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:13.684 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:13.576+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:13.685 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:13.577+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:14.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:13.741+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:13.741+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:13.741+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:13.742+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:13.682+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:13.682+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:13.682+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:14.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:13.683+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:14.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:14.721+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:14.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:14.721+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:14.721+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:14.722+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:14.707+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:14.707+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:14.707+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:14.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:14.708+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:14.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:14.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:14.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:14.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:14.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:14.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:14.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:14.551+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:15.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:15.715+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:15.715+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:15.715+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:15.716+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:15.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:15.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:15.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:15.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:15.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:15.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:15.556+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:15.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:15.672+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:15.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:15.672+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:15.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:15.672+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:15.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:15.673+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:16.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:16.631+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:16.631+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:16.631+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:16.631+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:16.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:16.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:16.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:16.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:16.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:16.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:16.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:16.671+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:16.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:16.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:16.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:16.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:16.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:16.551+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:16.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:16.552+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:17.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:26:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:26:17.483+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28101 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:26:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:17.629+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:17.629+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:17.629+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:17.629+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:17.622+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:17.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:17.622+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:17.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:17.622+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:17.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:17.623+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:17.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:17.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:17.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:17.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:17.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:17.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:17.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:17.568+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:18.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:18.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:18.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:18.589+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:18.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:18.628+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:18.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:18.628+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:18.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:18.628+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:18.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:18.629+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:18.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:18.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:18.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:18.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:18.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:18.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:18.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:18.609+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:19.674+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:19.674+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:19.674+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:19.675+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:19.583+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:19.583+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:19.583+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:19.584+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:19.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:19.562+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:19.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:19.562+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:19.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:19.562+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:19.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:19.562+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:20.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:20.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:20.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:20.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:20.588+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:20.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:20.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:20.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:20.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:20.661+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:20.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:20.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:20.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:20.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:20.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:20.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:20.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:20.516+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:21.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:21.641+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:21.641+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:21.641+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:21.641+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:21.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:21.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:21.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:21.480+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:21.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:21.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:21.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:21.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:21.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:21.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:21.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:21.590+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:22.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:26:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:26:22.486+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28160 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:26:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:22.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:22.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:22.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:22.638+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:22.508+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:22.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:22.508+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:22.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:22.508+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:22.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:22.508+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:22.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:22.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:22.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:22.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:22.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:22.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:22.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:22.543+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:23.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:23.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:23.504+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:23.504+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:23.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:23.560+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:23.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:23.560+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:23.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:23.560+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:23.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:23.560+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:24.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:23.661+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:24.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:23.661+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:23.661+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:23.662+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:24.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:24.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:24.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:24.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:24.697+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:24.577+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:24.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:24.577+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:24.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:24.577+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:24.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:24.577+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:24.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:24.506+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:24.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:24.506+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:24.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:24.506+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:24.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:24.506+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:25.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:25.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:25.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:25.501+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:25.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:25.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:25.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:25.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:25.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:25.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:25.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:25.711+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:25.574+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:25.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:25.574+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:25.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:25.574+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:25.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:25.575+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:26.760 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:26.519+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:26.760 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:26.519+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:26.761 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:26.519+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:26.761 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:26.520+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:26.761 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:26.556+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:26.761 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:26.556+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:26.761 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:26.556+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:26.761 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:26.556+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:27.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:26.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:27.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:26.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:26.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:26.760+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:27.765 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:27.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:27.765 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:27.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:27.765 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:27.503+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:27.765 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:27.503+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:27.766 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:27.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:27.766 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:27.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:27.766 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:27.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:27.766 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:27.528+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:27.766 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:26:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:26:27.489+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28219 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:26:28.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:27.763+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:27.763+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:27.763+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:27.764+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:28.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:28.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:28.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:28.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:28.491+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:28.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:28.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:28.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:28.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:28.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:28.720+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:28.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:28.546+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:28.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:28.546+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:28.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:28.546+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:28.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:28.547+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:29.713 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:29.454+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:29.713 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:29.454+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:29.713 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:29.454+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:29.713 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:29.455+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:29.714 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:29.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:29.714 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:29.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:29.714 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:29.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:29.714 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:29.566+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:29.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:29.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:29.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:29.712+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:30.702 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:30.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:30.702 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:30.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:30.702 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:30.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:30.702 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:30.604+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:30.702 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:30.445+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:30.703 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:30.445+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:30.703 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:30.445+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:30.703 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:30.446+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:30.700+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:30.700+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:30.700+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:30.701+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:31.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:31.478+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:31.478+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:31.478+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:31.479+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:31.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:31.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:31.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:31.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:31.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:31.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:31.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:31.617+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:31.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:31.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:31.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:31.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:31.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:31.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:31.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:31.668+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:32.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:26:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:26:32.491+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28277 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:26:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:32.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:32.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:32.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:32.609+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:32.694+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:32.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:32.694+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:32.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:32.694+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:32.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:32.695+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:32.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:32.487+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:32.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:32.487+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:32.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:32.487+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:32.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:32.488+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:33.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:33.576+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:33.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:33.576+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:33.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:33.576+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:33.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:33.577+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:33.730 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:33.459+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:33.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:33.459+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:33.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:33.459+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:33.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:33.460+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:33.731 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:33.703+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:33.731 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:33.703+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:33.731 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:33.703+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:33.732 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:33.704+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:34.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:34.692+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:34.692+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:34.692+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:34.692+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:34.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:34.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:34.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:34.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:34.479+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:34.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:34.579+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:34.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:34.579+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:34.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:34.579+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:34.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:34.580+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:35.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:35.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:35.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:35.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:35.718+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:35.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:35.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:35.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:35.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:35.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:35.521+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:35.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:35.567+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:35.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:35.567+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:35.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:35.567+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:35.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:35.567+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:36.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:36.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:36.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:36.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:36.593+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:36.674+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:36.674+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:36.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:36.674+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:36.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:36.674+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:36.493+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:36.493+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:36.493+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:36.495+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:37.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:37.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:37.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:37.553+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:37.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:37.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:37.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:37.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:37.668+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:37.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:37.493+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:37.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:37.493+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:37.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:37.493+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:37.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:37.494+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:37.833 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:26:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:26:37.493+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28336 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:26:38.822 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:38.531+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:38.823 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:38.531+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:38.823 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:38.531+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:38.823 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:38.532+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:38.824 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:38.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:38.824 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:38.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:38.824 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:38.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:38.824 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:38.622+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:38.825 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:38.519+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:38.825 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:38.519+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:38.825 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:38.519+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:38.826 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:38.520+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:39.565+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:39.565+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:39.565+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:39.566+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:39.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:39.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:39.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:39.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:39.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:39.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:39.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:39.656+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:39.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:39.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:39.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:39.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:39.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:39.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:39.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:39.506+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:40.576+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:40.576+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:40.576+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:40.577+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:40.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:40.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:40.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:40.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:40.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:40.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:40.688+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:40.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:40.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:40.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:40.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:40.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:40.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:40.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:40.550+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:41.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:41.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:41.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:41.554+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:41.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:41.695+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:41.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:41.695+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:41.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:41.695+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:41.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:41.696+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:41.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:41.570+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:41.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:41.570+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:41.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:41.570+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:41.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:41.570+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:42.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:42.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:42.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:42.734+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:42.735+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:42.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:42.576+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:42.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:42.576+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:42.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:42.576+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:42.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:42.577+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:42.832 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:26:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:26:42.497+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28395 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:26:42.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:42.585+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:42.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:42.585+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:42.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:42.585+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:42.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:42.586+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:43.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:43.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:43.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:43.562+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:43.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:43.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:43.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:43.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:43.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:43.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:43.737+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:43.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:43.626+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:43.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:43.626+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:43.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:43.626+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:43.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:43.627+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:44.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:44.563+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:44.563+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:44.563+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:44.564+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:44.649+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:44.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:44.649+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:44.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:44.649+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:44.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:44.650+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:44.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:44.691+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:44.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:44.691+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:44.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:44.691+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:44.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:44.692+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:45.611 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:45.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:45.612 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:45.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:45.612 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:45.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:45.612 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:45.557+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:45.668+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:46.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:45.668+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:46.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:45.668+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:46.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:45.669+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:46.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:45.610+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:46.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:45.610+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:46.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:45.610+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:46.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:45.611+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:46.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:46.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:46.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:46.599+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:47.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:46.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:47.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:46.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:47.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:46.717+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:47.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:46.718+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:47.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:46.652+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:47.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:46.652+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:47.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:46.652+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:47.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:46.653+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:47.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:26:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:26:47.500+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28451 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:26:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:47.743+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:47.743+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:47.743+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:47.744+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:47.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:47.550+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:47.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:47.550+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:47.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:47.550+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:47.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:47.550+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:47.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:47.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:47.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:47.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:47.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:47.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:47.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:47.625+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:48.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:48.699+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:48.699+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:48.699+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:48.700+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:48.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:48.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:48.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:48.556+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:48.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:48.646+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:48.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:48.646+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:48.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:48.646+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:48.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:48.646+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:49.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:49.623+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:49.623+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:49.623+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:49.624+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:49.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:49.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:49.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:49.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:49.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:49.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:49.523+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:49.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:49.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:49.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:49.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:49.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:49.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:49.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:49.670+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:50.692+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:50.692+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:50.692+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:50.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:50.693+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:50.670+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:50.670+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:50.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:50.670+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:50.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:50.671+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:50.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:50.487+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:50.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:50.487+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:50.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:50.487+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:50.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:50.488+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:51.810 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:51.661+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:51.810 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:51.661+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:51.811 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:51.661+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:51.811 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:51.662+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:51.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:51.625+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:51.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:51.625+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:51.811 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:51.625+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:51.812 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:51.626+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:51.812 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:51.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:51.812 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:51.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:51.812 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:51.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:51.812 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:51.515+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:52.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:26:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:26:52.503+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28470 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:26:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:52.646+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:52.646+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:52.646+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:52.647+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:52.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:52.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:52.479+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:52.479+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:52.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:52.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:52.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:52.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:52.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:52.625+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:52.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:52.626+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:53.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:53.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:53.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:53.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:53.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:53.611+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:53.634+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:53.634+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:53.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:53.634+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:53.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:53.635+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:53.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:53.508+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:53.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:53.508+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:53.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:53.508+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:53.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:53.509+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:54.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:54.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:54.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:54.597+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:54.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:54.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:54.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:54.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:54.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:54.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:54.620+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:54.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:54.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:54.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:54.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:54.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:54.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:54.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:54.481+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:55.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:55.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:55.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:55.621+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:55.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:55.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:55.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:55.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:55.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:55.617+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:55.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:55.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:55.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:55.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:55.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:55.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:55.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:55.527+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:56.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:56.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:56.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:56.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:56.605+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:56.562+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:56.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:56.562+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:56.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:56.562+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:56.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:56.563+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:56.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:56.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:56.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:56.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:56.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:56.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:56.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:56.661+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:57.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:57.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:57.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:57.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:57.597+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:57.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:57.646+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:57.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:57.646+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:57.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:57.646+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:57.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:57.646+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:57.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:57.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:57.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:57.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:57.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:57.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:57.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:57.587+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:57.833 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:26:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:26:57.505+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28484 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:26:58.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:58.688+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:58.688+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:58.688+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:58.689+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:58.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:58.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:58.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:58.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:58.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:58.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:58.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:58.589+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:26:58.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:58.544+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:58.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:58.544+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:58.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:58.544+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:58.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:58.545+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:59.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:59.676+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:59.676+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:59.676+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:26:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:26:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:26:59.677+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:59.552+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:59.552+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:59.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:59.552+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:26:59.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:26:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:26:59.553+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:26:59.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:59.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:59.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:59.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:59.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:59.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:26:59.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:26:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:26:59.546+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:00.531+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:00.531+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:00.531+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:00.532+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:00.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:00.635+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:00.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:00.635+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:00.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:00.635+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:00.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:00.636+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:00.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:00.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:00.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:00.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:00.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:00.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:00.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:00.596+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:01.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:01.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:01.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:01.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:01.610+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:01.517+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:01.517+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:01.517+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:01.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:01.518+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:01.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:01.615+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:01.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:01.615+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:01.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:01.615+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:01.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:01.616+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:02.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:27:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:27:02.509+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28498 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:27:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:02.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:02.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:02.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:02.629+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:02.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:02.633+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:02.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:02.633+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:02.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:02.633+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:02.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:02.633+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:02.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:02.501+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:02.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:02.501+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:02.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:02.501+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:02.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:02.502+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:03.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:03.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:03.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:03.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:03.586+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:03.640+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:03.640+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:03.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:03.640+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:03.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:03.641+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:03.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:03.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:03.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:03.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:03.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:03.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:03.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:03.520+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:04.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:04.558+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:04.558+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:04.558+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:04.558+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:04.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:04.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:04.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:04.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:04.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:04.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:04.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:04.530+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:04.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:04.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:04.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:04.608+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:05.592+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:05.592+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:05.592+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:05.593+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:05.507+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:05.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:05.507+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:05.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:05.507+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:05.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:05.507+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:05.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:05.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:05.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:05.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:05.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:05.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:05.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:05.599+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:06.605 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:06.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:06.606 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:06.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:06.606 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:06.566+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:06.606 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:06.567+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:06.606 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:06.540+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:06.606 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:06.540+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:06.606 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:06.540+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:06.607 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:06.541+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:07.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:06.604+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:06.604+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:06.604+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:06.604+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:07.519+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:07.519+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:07.519+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:07.520+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:07.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:07.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:07.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:07.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:07.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:07.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:07.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:07.646+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:07.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:07.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:07.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:07.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:07.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:07.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:07.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:07.503+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:07.833 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:27:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:27:07.512+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28512 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:27:08.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:08.537+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:08.537+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:08.537+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:08.537+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:08.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:08.563+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:08.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:08.563+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:08.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:08.563+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:08.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:08.564+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:08.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:08.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:08.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:08.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:08.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:08.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:08.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:08.672+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:09.714 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:09.554+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:09.714 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:09.554+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:09.714 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:09.554+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:09.715 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:09.554+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:09.715 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:09.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:09.715 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:09.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:09.715 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:09.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:09.715 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:09.507+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:10.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:09.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:09.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:09.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:09.712+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:10.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:10.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:10.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:10.711+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:10.711+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:10.485+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:10.485+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:10.485+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:10.486+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:10.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:10.580+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:10.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:10.580+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:10.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:10.580+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:10.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:10.581+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:11.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:11.690+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:11.690+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:11.690+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:11.690+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:11.600+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:11.600+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:11.600+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:11.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:11.600+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:11.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:11.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:11.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:11.490+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:12.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:27:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:27:12.514+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28525 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:27:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:12.531+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:12.531+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:12.531+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:12.532+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:12.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:12.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:12.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:12.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:12.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:12.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:12.592+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:12.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:12.718+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:12.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:12.718+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:12.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:12.718+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:12.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:12.718+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:14.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:13.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:13.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:13.621+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:13.621+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:13.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:14.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:13.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:14.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:13.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:14.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:13.582+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:14.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:13.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:14.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:13.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:14.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:13.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:14.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:13.679+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:14.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:14.641+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:14.641+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:14.641+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:14.641+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:14.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:14.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:14.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:14.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:14.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:14.683+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:14.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:14.621+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:14.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:14.621+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:14.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:14.621+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:14.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:14.621+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:15.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:15.678+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:15.678+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:15.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:15.679+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:15.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:15.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:15.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:15.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:15.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:15.689+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:15.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:15.579+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:15.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:15.579+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:15.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:15.579+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:15.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:15.579+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:17.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:16.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:17.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:16.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:17.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:16.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:17.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:16.703+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:16.681+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:16.681+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:17.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:16.681+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:17.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:16.681+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:17.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:16.600+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:17.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:16.600+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:17.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:16.600+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:17.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:16.601+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:17.832 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:27:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:27:17.517+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28539 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:27:17.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:17.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:17.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:17.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:17.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:17.711+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:17.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:17.711+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:17.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:17.643+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:17.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:17.643+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:17.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:17.643+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:17.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:17.643+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:17.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:17.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:17.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:17.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:17.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:17.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:17.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:17.696+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:18.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:18.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:18.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:18.710+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:18.632+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:18.632+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:19.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:18.632+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:19.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:18.632+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:18.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:18.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:18.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:18.709+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:20.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:19.705+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:19.705+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:19.705+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:19.705+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:20.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:19.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:20.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:19.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:20.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:19.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:20.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:19.747+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:20.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:19.594+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:20.084 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:19.594+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:20.084 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:19.594+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:20.084 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:19.595+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:20.712+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:20.712+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:20.712+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:20.712+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:20.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:20.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:20.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:20.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:20.769+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:20.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:20.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:20.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:20.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:20.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:20.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:20.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:20.578+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:21.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:21.804+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:21.804+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:21.804+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:21.804+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:21.564+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:21.564+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:21.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:21.564+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:21.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:21.565+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:21.728+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:21.728+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:21.728+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:21.729+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:22.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:27:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:27:22.521+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28553 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:27:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:22.612+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:22.612+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:22.612+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:22.612+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:22.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:22.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:22.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:22.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:22.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:22.689+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:23.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:22.839+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:23.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:22.839+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:22.839+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:22.840+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:23.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:23.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:23.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:23.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:23.651+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:23.657+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:23.657+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:23.657+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:23.658+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:24.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:23.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:24.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:23.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:23.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:23.872+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:24.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:24.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:24.895 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:24.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:24.895 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:24.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:24.895 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:24.621+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:24.895 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:24.645+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:24.895 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:24.645+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:24.895 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:24.645+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:24.896 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:24.646+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:25.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:24.893+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:24.893+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:24.893+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:24.894+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:25.937 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:25.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:25.938 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:25.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:25.938 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:25.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:25.938 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:25.618+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:25.938 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:25.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:25.938 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:25.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:25.938 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:25.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:25.939 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:25.610+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:26.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:25.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:25.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:25.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:25.936+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:26.949 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:26.632+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:26.950 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:26.632+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:26.950 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:26.632+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:26.950 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:26.632+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:26.950 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:26.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:26.950 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:26.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:26.950 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:26.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:26.951 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:26.638+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:27.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:26.948+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:26.948+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:26.948+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:26.948+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:27.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:27:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:27:27.524+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28567 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:27:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:27.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:27.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:27.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:27.626+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:27.640+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:27.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:27.640+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:27.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:27.640+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:27.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:27.640+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:28.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:27.942+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:27.942+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:27.942+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:27.943+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:28.950 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:28.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:28.950 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:28.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:28.950 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:28.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:28.950 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:28.672+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:28.950 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:28.631+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:28.951 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:28.631+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:28.951 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:28.631+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:28.951 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:28.631+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:29.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:28.948+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:28.948+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:28.948+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:28.949+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:29.880 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:29.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:29.880 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:29.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:29.880 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:29.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:29.880 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:29.621+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:29.881 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:29.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:29.881 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:29.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:29.881 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:29.625+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:29.881 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:29.625+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:30.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:29.986+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:29.986+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:29.986+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:29.986+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:30.913 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:30.635+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:30.913 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:30.635+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:30.914 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:30.635+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:30.914 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:30.636+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:30.914 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:30.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:30.914 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:30.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:30.915 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:30.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:30.915 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:30.589+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:31.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:31.031+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:31.031+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:31.031+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:31.032+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:32.064 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:31.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:32.065 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:31.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:32.065 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:31.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:32.065 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:31.614+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:32.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:31.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:32.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:31.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:32.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:31.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:32.066 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:31.617+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:32.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:32.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:32.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:32.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:32.063+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:32.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:32.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:32.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:32.657+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:32.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:32.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:32.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:32.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:32.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:32.582+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:32.832 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:27:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:27:32.527+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28580 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:27:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:33.068+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:33.068+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:33.068+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:33.068+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:33.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:33.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:33.678+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:33.679+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:33.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:33.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:33.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:33.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:33.555+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:34.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:34.113+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:34.113+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:34.113+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:34.114+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:34.975 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:34.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:34.975 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:34.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:34.976 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:34.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:34.976 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:34.582+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:34.976 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:34.724+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:34.976 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:34.724+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:34.976 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:34.724+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:34.976 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:34.724+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:35.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:35.144+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:35.144+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:35.144+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:35.145+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:35.766+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:35.766+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:35.766+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:35.767+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:35.552+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:35.552+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:35.552+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:35.553+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:36.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:36.125+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:36.125+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:36.125+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:36.126+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:37.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:36.790+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:37.057 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:36.790+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:37.058 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:36.790+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:37.058 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:36.791+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:37.058 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:36.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:37.058 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:36.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:37.058 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:36.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:37.058 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:36.589+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:37.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:37.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:37.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:37.080+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:37.081+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:37.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:37.571+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:37.571+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:37.571+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:37.572+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:37.762+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:37.762+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:37.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:37.762+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:37.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:37.762+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:37.832 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:27:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:27:37.530+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28594 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:27:38.401 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:38.118+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:38.401 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:38.118+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:38.402 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:38.118+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:38.402 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:38.119+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:39.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:38.612+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:38.612+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:38.612+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:38.613+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:39.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:38.724+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:39.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:38.724+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:39.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:38.724+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:39.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:38.725+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:39.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:39.134+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:39.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:39.134+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:39.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:39.134+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:39.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:39.135+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:40.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:39.707+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:39.707+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:39.707+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:39.708+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:40.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:39.662+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:40.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:39.662+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:40.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:39.662+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:40.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:39.663+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:40.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:40.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:40.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:40.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:40.089+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:41.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:40.704+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:41.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:40.704+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:41.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:40.704+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:41.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:40.705+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:40.756+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:40.756+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:41.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:40.756+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:41.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:40.757+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:41.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:41.081+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:41.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:41.081+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:41.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:41.081+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:41.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:41.082+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:42.051 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:41.738+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:42.051 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:41.738+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:42.052 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:41.738+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:42.052 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:41.739+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:42.052 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:41.670+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:42.052 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:41.670+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:42.052 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:41.670+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:42.052 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:41.671+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:42.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:42.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:42.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:42.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:42.051+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:42.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:27:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:27:42.532+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28608 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:27:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:42.770+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:42.770+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:42.770+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:42.771+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:42.629+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:42.629+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:42.629+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:42.630+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:43.076+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:43.076+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:43.076+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:43.077+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:44.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:43.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:43.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:43.787+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:43.788+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:44.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:43.625+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:44.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:43.625+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:44.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:43.625+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:44.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:43.626+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:44.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:44.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:44.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:44.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:44.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:44.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:44.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:44.123+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:45.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:44.825+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:45.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:44.825+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:45.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:44.825+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:45.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:44.826+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:45.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:44.648+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:45.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:44.648+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:45.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:44.648+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:45.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:44.649+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:45.318 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-10T04:27:45.315+0000 7f2e39751640 0 monclient(hunting): authenticate timed out after 300 2024-09-10T04:27:45.318 INFO:teuthology.orchestra.run.smithi112.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-09-10T04:27:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:45.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:45.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:45.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:45.162+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:46.029 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:45.662+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:46.029 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:45.662+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:46.029 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:45.662+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:46.029 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:45.663+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:46.030 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:45.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:46.030 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:45.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:46.030 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:45.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:46.030 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:45.862+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:46.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T04:27:46.058 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-09-10T04:27:46.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T04:27:46.058 DEBUG:teuthology.orchestra.run.smithi112:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-09-10T04:27:46.102 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/smack: Cannot open: Input/output error 2024-09-10T04:27:46.102 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/p9auth: Cannot open: Input/output error 2024-09-10T04:27:46.103 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/tpm_tools: Cannot open: Input/output error 2024-09-10T04:27:46.103 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/cap_bounds: Cannot open: Input/output error 2024-09-10T04:27:46.103 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/rpctirpc: Cannot open: Input/output error 2024-09-10T04:27:46.103 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/mm: Cannot open: Input/output error 2024-09-10T04:27:46.103 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/network_stress.appl: Cannot open: Input/output error 2024-09-10T04:27:46.103 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/network_stress.selected: Cannot open: Input/output error 2024-09-10T04:27:46.103 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/tcp_cmds: Cannot open: Input/output error 2024-09-10T04:27:46.104 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/io_cd: Cannot open: Input/output error 2024-09-10T04:27:46.104 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/fsx: Cannot open: Input/output error 2024-09-10T04:27:46.104 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/ima: Cannot open: Input/output error 2024-09-10T04:27:46.104 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/multicast: Cannot open: Input/output error 2024-09-10T04:27:46.104 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/can: Cannot open: Input/output error 2024-09-10T04:27:46.104 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/numa: Cannot open: Input/output error 2024-09-10T04:27:46.104 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/ipv6: Cannot open: Input/output error 2024-09-10T04:27:46.104 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/ltp-aiodio.part4: Cannot open: Input/output error 2024-09-10T04:27:46.105 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/ltp-aio-stress.part1: Cannot open: Input/output error 2024-09-10T04:27:46.105 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/ipv6_noexpect: Cannot open: Input/output error 2024-09-10T04:27:46.105 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/stress.part1: Cannot open: Input/output error 2024-09-10T04:27:46.105 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/fs: Cannot open: Input/output error 2024-09-10T04:27:46.105 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/sched: Cannot open: Input/output error 2024-09-10T04:27:46.105 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/network_stress.udp: Cannot open: Input/output error 2024-09-10T04:27:46.105 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/containers: Cannot open: Input/output error 2024-09-10T04:27:46.105 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/io: Cannot open: Input/output error 2024-09-10T04:27:46.106 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/tcp_cmds_expect: Cannot open: Input/output error 2024-09-10T04:27:46.106 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/tirpc: Cannot open: Input/output error 2024-09-10T04:27:46.106 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/Makefile: Cannot open: Input/output error 2024-09-10T04:27:46.106 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/network_stress.tcp: Cannot open: Input/output error 2024-09-10T04:27:46.106 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/io_floppy: Cannot open: Input/output error 2024-09-10T04:27:46.106 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/filecaps: Cannot open: Input/output error 2024-09-10T04:27:46.106 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/ltp-aiodio.part1: Cannot open: Input/output error 2024-09-10T04:27:46.106 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/stress.part2: Cannot open: Input/output error 2024-09-10T04:27:46.106 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/crashme: Cannot open: Input/output error 2024-09-10T04:27:46.107 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/rpc: Cannot open: Input/output error 2024-09-10T04:27:46.107 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/fs_ext4: Cannot open: Input/output error 2024-09-10T04:27:46.107 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/test_dma_thread_diotest7: Cannot open: Input/output error 2024-09-10T04:27:46.107 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/lvm.part2: Cannot open: Input/output error 2024-09-10T04:27:46.107 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/ltp-aiodio.part3: Cannot open: Input/output error 2024-09-10T04:27:46.107 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/tcp_cmds_noexpect: Cannot open: Input/output error 2024-09-10T04:27:46.107 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/ipv6_expect: Cannot open: Input/output error 2024-09-10T04:27:46.107 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/fs_readonly: Cannot open: Input/output error 2024-09-10T04:27:46.108 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/power_management_tests: Cannot open: Input/output error 2024-09-10T04:27:46.108 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/ltp-aiodio.part2: Cannot open: Input/output error 2024-09-10T04:27:46.108 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/ipv6_lib: Cannot open: Input/output error 2024-09-10T04:27:46.108 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/nptl: Cannot open: Input/output error 2024-09-10T04:27:46.108 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/network_stress.multicast: Cannot open: Input/output error 2024-09-10T04:27:46.108 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/connectors: Cannot open: Input/output error 2024-09-10T04:27:46.108 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/nw_under_ns: Cannot open: Input/output error 2024-09-10T04:27:46.109 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/selinux: Cannot open: Input/output error 2024-09-10T04:27:46.109 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/tcp_cmds_addition: Cannot open: Input/output error 2024-09-10T04:27:46.109 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/ltplite: Cannot open: Input/output error 2024-09-10T04:27:46.109 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/controllers: Cannot open: Input/output error 2024-09-10T04:27:46.109 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/cpuacct: Cannot open: Input/output error 2024-09-10T04:27:46.109 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/quickhit: Cannot open: Input/output error 2024-09-10T04:27:46.109 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/perfcounters: Cannot open: Input/output error 2024-09-10T04:27:46.109 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/network_stress.route: Cannot open: Input/output error 2024-09-10T04:27:46.110 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/hugetlb: Cannot open: Input/output error 2024-09-10T04:27:46.110 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/network_stress.interface: Cannot open: Input/output error 2024-09-10T04:27:46.110 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/cpuhotplug: Cannot open: Input/output error 2024-09-10T04:27:46.111 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/admin_tools: Cannot open: Input/output error 2024-09-10T04:27:46.111 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/scsi_debug.part1: Cannot open: Input/output error 2024-09-10T04:27:46.111 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/modules: Cannot open: Input/output error 2024-09-10T04:27:46.111 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/stress.part3: Cannot open: Input/output error 2024-09-10T04:27:46.111 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/syscalls: Cannot open: Input/output error 2024-09-10T04:27:46.111 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/ipc: Cannot open: Input/output error 2024-09-10T04:27:46.111 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/network_stress.icmp: Cannot open: Input/output error 2024-09-10T04:27:46.112 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/hyperthreading: Cannot open: Input/output error 2024-09-10T04:27:46.112 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/sctp: Cannot open: Input/output error 2024-09-10T04:27:46.112 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/timers: Cannot open: Input/output error 2024-09-10T04:27:46.112 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/pty: Cannot open: Input/output error 2024-09-10T04:27:46.112 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/lvm.part1: Cannot open: Input/output error 2024-09-10T04:27:46.112 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/network_stress.broken_ip: Cannot open: Input/output error 2024-09-10T04:27:46.112 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/network_stress.whole: Cannot open: Input/output error 2024-09-10T04:27:46.112 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/fs_bind: Cannot open: Input/output error 2024-09-10T04:27:46.113 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/network_commands: Cannot open: Input/output error 2024-09-10T04:27:46.113 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/power_management_tests_exclusive: Cannot open: Input/output error 2024-09-10T04:27:46.113 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/nfs: Cannot open: Input/output error 2024-09-10T04:27:46.113 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest/pipes: Cannot open: Input/output error 2024-09-10T04:27:46.113 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runtest: Cannot utime: Input/output error 2024-09-10T04:27:46.113 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc: Cannot mkdir: Input/output error 2024-09-10T04:27:46.113 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3: Cannot mkdir: Input/output error 2024-09-10T04:27:46.113 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/random_range.3: Cannot open: Input/output error 2024-09-10T04:27:46.114 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/str_to_bytes.3: Cannot open: Input/output error 2024-09-10T04:27:46.114 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/usctest.3: Cannot open: Input/output error 2024-09-10T04:27:46.114 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/string_to_tokens.3: Cannot open: Input/output error 2024-09-10T04:27:46.114 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/Makefile: Cannot open: Input/output error 2024-09-10T04:27:46.114 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/pattern.3: Cannot open: Input/output error 2024-09-10T04:27:46.114 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/tst_tmpdir.3: Cannot open: Input/output error 2024-09-10T04:27:46.114 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/forker.3: Cannot open: Input/output error 2024-09-10T04:27:46.114 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/rmobj.3: Cannot open: Input/output error 2024-09-10T04:27:46.114 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/random_range_seed.3: Cannot open: Input/output error 2024-09-10T04:27:46.115 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/parse_open_flags.3: Cannot open: Input/output error 2024-09-10T04:27:46.115 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/parse_opts.3: Cannot open: Input/output error 2024-09-10T04:27:46.115 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/write_log.3: Cannot open: Input/output error 2024-09-10T04:27:46.115 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/parse_ranges.3: Cannot open: Input/output error 2024-09-10T04:27:46.115 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/tst_set_error.3: Cannot open: Input/output error 2024-09-10T04:27:46.116 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/tst_res.3: Cannot open: Input/output error 2024-09-10T04:27:46.116 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/tst_sig.3: Cannot open: Input/output error 2024-09-10T04:27:46.116 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi112.front.sepia.ceph.com 2024-09-10T04:27:46.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-10T04:27:46.117 DEBUG:teuthology.orchestra.run.smithi112:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-10T04:27:46.123 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T04:27:46.124 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man3/get_attrib.3: Cannot open: Input/output error 2024-09-10T04:27:46.124 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man1: Cannot mkdir: Input/output error 2024-09-10T04:27:46.124 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man1/ltp-bump.1: Cannot open: Input/output error 2024-09-10T04:27:46.124 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man1/doio.1: Cannot open: Input/output error 2024-09-10T04:27:46.124 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man1/Makefile: Cannot open: Input/output error 2024-09-10T04:27:46.124 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man1/ltp-pan.1: Cannot open: Input/output error 2024-09-10T04:27:46.124 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/man1/iogen.1: Cannot open: Input/output error 2024-09-10T04:27:46.124 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/testcases: Cannot mkdir: Input/output error 2024-09-10T04:27:46.125 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/testcases/kernel.txt: Cannot open: Input/output error 2024-09-10T04:27:46.125 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/testcases/misc.txt: Cannot open: Input/output error 2024-09-10T04:27:46.125 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/testcases/network.txt: Cannot open: Input/output error 2024-09-10T04:27:46.125 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/testcases/commands.txt: Cannot open: Input/output error 2024-09-10T04:27:46.125 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/testcases/open_posix_testsuite.txt: Cannot open: Input/output error 2024-09-10T04:27:46.125 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/testcases/open_hpi_testsuite.txt: Cannot open: Input/output error 2024-09-10T04:27:46.125 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/ltp-howto.lyx: Cannot open: Input/output error 2024-09-10T04:27:46.125 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/Makefile: Cannot open: Input/output error 2024-09-10T04:27:46.126 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/automation-cookbook.html: Cannot open: Input/output error 2024-09-10T04:27:46.126 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/examples: Cannot mkdir: Input/output error 2024-09-10T04:27:46.126 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/examples/quickhit.c: Cannot open: Input/output error 2024-09-10T04:27:46.126 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/examples/sudoers: Cannot open: Input/output error 2024-09-10T04:27:46.126 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/examples/super.tab: Cannot open: Input/output error 2024-09-10T04:27:46.126 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/ltp-run-files.txt: Cannot open: Input/output error 2024-09-10T04:27:46.126 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/tpm-commands.txt: Cannot open: Input/output error 2024-09-10T04:27:46.126 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/ltp-howto.txt: Cannot open: Input/output error 2024-09-10T04:27:46.127 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/network_stress.txt: Cannot open: Input/output error 2024-09-10T04:27:46.127 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/doc/automation-README.html: Cannot open: Input/output error 2024-09-10T04:27:46.127 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4: Cannot mkdir: Input/output error 2024-09-10T04:27:46.127 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4/ltp-cgroupstats.m4: Cannot open: Input/output error 2024-09-10T04:27:46.127 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4/ltp-signalfd.m4: Cannot open: Input/output error 2024-09-10T04:27:46.127 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4/GNUmakefile: Cannot open: Input/output error 2024-09-10T04:27:46.127 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4/ltp-eventfd.m4: Cannot open: Input/output error 2024-09-10T04:27:46.127 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4/ltp-quota.m4: Cannot open: Input/output error 2024-09-10T04:27:46.127 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4/ltp-version.m4: Cannot open: Input/output error 2024-09-10T04:27:46.128 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4/ltp-cap.m4: Cannot open: Input/output error 2024-09-10T04:27:46.128 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4/ltp-selinux.m4: Cannot open: Input/output error 2024-09-10T04:27:46.128 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4/ltp-signal.m4: Cannot open: Input/output error 2024-09-10T04:27:46.128 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4/ltp-ptrace.m4: Cannot open: Input/output error 2024-09-10T04:27:46.128 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4/ltp-numa.m4: Cannot open: Input/output error 2024-09-10T04:27:46.128 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4/Makefile.in: Cannot open: Input/output error 2024-09-10T04:27:46.128 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4/ltp-modify_ldt.m4: Cannot open: Input/output error 2024-09-10T04:27:46.128 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4/ltp-unshare.m4: Cannot open: Input/output error 2024-09-10T04:27:46.129 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4/ltp-taskstats.m4: Cannot open: Input/output error 2024-09-10T04:27:46.129 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4/ltp-crypto.m4: Cannot open: Input/output error 2024-09-10T04:27:46.129 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/m4/Makefile.am: Cannot open: Input/output error 2024-09-10T04:27:46.129 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/runltp: Cannot open: Input/output error 2024-09-10T04:27:46.129 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/Makefile: Cannot open: Input/output error 2024-09-10T04:27:46.129 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/configure: Cannot open: Input/output error 2024-09-10T04:27:46.129 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/config.guess: Cannot open: Input/output error 2024-09-10T04:27:46.129 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/execltp.in: Cannot open: Input/output error 2024-09-10T04:27:46.130 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib: Cannot mkdir: Input/output error 2024-09-10T04:27:46.130 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/rmobj.c: Cannot open: Input/output error 2024-09-10T04:27:46.130 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/datapid.c: Cannot open: Input/output error 2024-09-10T04:27:46.130 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/tst_kvercmp.c: Cannot open: Input/output error 2024-09-10T04:27:46.130 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/tst_sig.c: Cannot open: Input/output error 2024-09-10T04:27:46.130 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/tst_cwd_has_free.c: Cannot open: Input/output error 2024-09-10T04:27:46.130 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/parse_opts.c: Cannot open: Input/output error 2024-09-10T04:27:46.130 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/tlibio.c: Cannot open: Input/output error 2024-09-10T04:27:46.130 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/cloner.c: Cannot open: Input/output error 2024-09-10T04:27:46.131 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/system_specific_process_info.c: Cannot open: Input/output error 2024-09-10T04:27:46.131 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/tst_tmpdir.c: Cannot open: Input/output error 2024-09-10T04:27:46.131 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/open_flags.c: Cannot open: Input/output error 2024-09-10T04:27:46.131 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/get_high_address.c: Cannot open: Input/output error 2024-09-10T04:27:46.131 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/Makefile: Cannot open: Input/output error 2024-09-10T04:27:46.131 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/write_log.c: Cannot open: Input/output error 2024-09-10T04:27:46.131 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/file_lock.c: Cannot open: Input/output error 2024-09-10T04:27:46.131 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/tst_is_cwd.c: Cannot open: Input/output error 2024-09-10T04:27:46.131 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/string_to_tokens.c: Cannot open: Input/output error 2024-09-10T04:27:46.132 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/system_specific_hugepages_info.c: Cannot open: Input/output error 2024-09-10T04:27:46.132 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/str_to_bytes.c: Cannot open: Input/output error 2024-09-10T04:27:46.132 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/random_range.c: Cannot open: Input/output error 2024-09-10T04:27:46.132 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/libtestsuite.c: Cannot open 2024-09-10T04:27:46.132 INFO:tasks.workunit.client.0.smithi112.stderr:gzip: : Input/output error 2024-09-10T04:27:46.132 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/ltp.pc.in: Cannot open: Input/output error 2024-09-10T04:27:46.132 INFO:tasks.workunit.client.0.smithi112.stderr:stdin: Stale file handle 2024-09-10T04:27:46.132 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231/lib/tst_res.c: Cannot open: Input/output error 2024-09-10T04:27:46.133 INFO:tasks.workunit.client.0.smithi112.stderr:tar: Unexpected EOF in archive 2024-09-10T04:27:46.133 INFO:tasks.workunit.client.0.smithi112.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-09-10T04:27:46.133 INFO:tasks.workunit.client.0.smithi112.stderr:tar: Error is not recoverable: exiting now 2024-09-10T04:27:46.133 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-09-10T04:27:46.133 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-10T04:27:46.187 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-10T04:27:46.187 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-10T04:27:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:46.208+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:46.208+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:46.208+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:46.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:46.208+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:46.634+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:46.634+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:46.634+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:46.635+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:47.202 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:46.855+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:47.202 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:46.855+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:47.202 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:46.855+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:47.202 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:46.856+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:47.538 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:47.199+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:47.539 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:47.199+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:47.539 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:47.199+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:47.539 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:47.200+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:47.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:47.642+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:47.642+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:47.642+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:47.643+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:47.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:27:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:27:47.534+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28622 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:27:48.166 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:47.841+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:48.166 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:47.841+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:48.166 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:47.841+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:48.166 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:47.842+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:48.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:48.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:48.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:48.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:48.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:48.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:48.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:48.164+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:48.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:48.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:48.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:48.617+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:48.831+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:49.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:48.831+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:49.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:48.831+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:49.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:48.832+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:49.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:49.123+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:49.123+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:49.123+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:49.123+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:50.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:49.789+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:49.789+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:49.789+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:49.790+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:50.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:49.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:50.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:49.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:50.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:49.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:50.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:49.625+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:50.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:50.132+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:50.132+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:50.132+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:50.133+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:51.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:50.646+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:51.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:50.646+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:51.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:50.646+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:51.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:50.647+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:51.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:50.834+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:51.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:50.834+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:51.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:50.834+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:51.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:50.834+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:51.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:51.097+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:51.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:51.097+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:51.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:51.097+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:51.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:51.098+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:52.067 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:51.879+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:52.067 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:51.879+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:52.067 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:51.879+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:52.068 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:51.880+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:52.068 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:51.681+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:52.068 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:51.681+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:52.068 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:51.681+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:52.068 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:51.681+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:52.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:52.065+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:52.066+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:52.066+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:52.066+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:52.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:27:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:27:52.538+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28635 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:27:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:52.660+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:52.660+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:52.660+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:52.661+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:53.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:52.837+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:53.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:52.837+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:53.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:52.837+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:53.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:52.837+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:53.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:53.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:53.034+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:53.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:53.034+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:54.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:53.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:54.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:53.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:54.002 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:53.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:54.002 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:53.818+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:54.002 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:53.671+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:54.003 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:53.671+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:54.003 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:53.671+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:54.003 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:53.672+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:54.000+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:54.000+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:54.000+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:54.000+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:55.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:54.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:55.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:54.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:55.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:54.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:55.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:54.818+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:55.046 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:54.678+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:55.047 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:54.678+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:55.047 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:54.678+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:55.047 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:54.679+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:55.044+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:55.044+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:55.044+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:55.045+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:55.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:55.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:55.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:55.817+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:56.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:55.682+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:56.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:55.682+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:56.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:55.682+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:56.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:55.683+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:56.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:56.086+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:56.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:56.086+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:56.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:56.086+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:56.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:56.087+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:57.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:56.664+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:56.664+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:56.664+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:56.664+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:57.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:56.843+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:57.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:56.843+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:57.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:56.843+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:57.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:56.844+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:57.543 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:57.099+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:57.543 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:57.099+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:57.544 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:57.099+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:57.544 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:57.100+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:57.807 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:57.682+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:57.808 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:57.682+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:57.808 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:57.682+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:57.808 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:57.683+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:57.808 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:27:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:27:57.541+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28649 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:27:58.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:57.806+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:57.806+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:57.806+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:57.806+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:58.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:58.079+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:58.079+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:58.079+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:58.080+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:59.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:58.643+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:58.643+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:58.643+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:27:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:58.643+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:27:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:58.831+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:58.831+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:59.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:58.831+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:27:59.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:58.832+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:27:59.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:59.103+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:59.103+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:59.103+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:27:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:27:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:27:59.103+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:00.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:59.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:00.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:59.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:00.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:59.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:00.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:27:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:27:59.631+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:59.875+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:59.875+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:59.875+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:00.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:27:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:27:59.875+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:00.478 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:00.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:00.478 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:00.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:00.478 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:00.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:00.479 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:00.153+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:00.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:00.670+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:00.670+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:00.670+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:00.671+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:01.173 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:00.881+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:01.173 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:00.881+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:01.173 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:00.881+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:01.173 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:00.882+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:01.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:01.171+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:01.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:01.171+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:01.171+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:01.172+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:02.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:01.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:01.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:01.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:01.669+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:01.896+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:01.896+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:02.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:01.896+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:02.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:01.896+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:02.546 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:02.207+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:02.546 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:02.207+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:02.547 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:02.207+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:02.547 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:02.207+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:02.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:28:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:28:02.544+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28663 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:28:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:02.681+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:02.681+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:02.681+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:02.682+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:03.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:03.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:03.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:03.169+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:03.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:02.944+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:03.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:02.944+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:03.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:02.944+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:03.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:02.945+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:03.931 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:03.659+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:03.931 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:03.659+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:03.931 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:03.659+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:03.932 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:03.659+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:04.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:03.929+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:03.929+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:03.929+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:03.930+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:04.153+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:04.153+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:04.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:04.153+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:04.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:04.153+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:04.954 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:04.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:04.954 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:04.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:04.954 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:04.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:04.954 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:04.625+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:05.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:04.952+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:04.952+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:04.952+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:04.952+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:05.104+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:05.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:05.104+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:05.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:05.104+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:05.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:05.105+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:05.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:05.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:05.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:05.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:05.631+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:06.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:06.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:06.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:06.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:06.060+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:06.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:05.979+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:06.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:05.979+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:06.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:05.979+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:06.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:05.980+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:06.977 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:06.639+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:06.977 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:06.639+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:06.977 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:06.639+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:06.977 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:06.640+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:07.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:07.071+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:07.071+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:07.071+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:07.071+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:07.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:06.975+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:07.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:06.975+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:07.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:06.975+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:07.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:06.976+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:07.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:28:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:28:07.547+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28677 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:28:07.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:07.660+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:07.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:07.660+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:07.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:07.660+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:07.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:07.661+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:08.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:07.977+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:07.977+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:07.977+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:07.978+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:08.024+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:08.024+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:08.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:08.024+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:08.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:08.025+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:08.956 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:08.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:08.956 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:08.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:08.957 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:08.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:08.957 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:08.638+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:09.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:08.953+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:08.953+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:08.953+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:08.954+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:09.010+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:09.010+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:09.010+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:09.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:09.011+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:09.983 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:09.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:09.983 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:09.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:09.983 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:09.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:09.983 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:09.631+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:10.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:09.991+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:10.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:09.991+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:10.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:09.991+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:10.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:09.992+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:10.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:09.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:10.334 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:09.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:10.334 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:09.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:10.334 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:09.982+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:10.988 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:10.672+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:10.989 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:10.672+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:10.989 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:10.672+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:10.990 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:10.673+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:10.987+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:10.987+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:10.987+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:10.988+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:11.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:11.020+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:11.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:11.021+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:11.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:11.021+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:11.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:11.021+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:12.020 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:11.695+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:12.020 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:11.695+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:12.020 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:11.695+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:12.021 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:11.696+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:12.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:12.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:12.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:12.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:12.018+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:12.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:12.065+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:12.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:12.065+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:12.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:12.065+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:12.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:12.066+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:12.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:12.722+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:12.722+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:12.722+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:12.723+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:12.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:28:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:28:12.549+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28690 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:28:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:13.003+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:13.003+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:13.003+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:13.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:13.004+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:13.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:13.114+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:13.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:13.114+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:13.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:13.114+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:13.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:13.114+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:13.998 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:13.679+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:13.998 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:13.679+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:13.998 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:13.679+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:13.998 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:13.680+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:14.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:13.996+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:13.996+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:13.996+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:13.997+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:14.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:14.141+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:14.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:14.141+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:14.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:14.141+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:14.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:14.142+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:15.023 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:14.647+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:15.023 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:14.647+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:15.023 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:14.647+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:15.023 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:14.648+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:15.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:15.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:15.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:15.022+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:15.101+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:15.101+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:15.101+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:15.102+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:16.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:15.672+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:16.035 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:15.672+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:16.035 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:15.672+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:16.035 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:15.673+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:16.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:16.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:16.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:16.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:16.061+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:16.092+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:16.092+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:16.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:16.092+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:16.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:16.092+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:16.680+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:16.680+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:16.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:16.680+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:16.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:16.681+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:17.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:17.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:17.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:17.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:17.065+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:17.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:17.136+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:17.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:17.136+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:17.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:17.136+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:17.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:17.137+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:17.682 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:28:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:28:17.552+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28704 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:28:18.024 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:17.680+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:18.024 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:17.680+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:18.024 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:17.680+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:18.024 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:17.681+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:18.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:18.187+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:18.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:18.187+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:18.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:18.187+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:18.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:18.188+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:18.022+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:18.022+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:18.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:18.022+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:18.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:18.023+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:19.031 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:18.714+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:19.031 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:18.714+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:19.032 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:18.714+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:19.032 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:18.715+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:19.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:19.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:19.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:19.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:19.030+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:19.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:19.181+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:19.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:19.181+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:19.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:19.181+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:19.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:19.182+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:20.064 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:19.691+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:20.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:19.691+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:20.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:19.691+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:20.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:19.692+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:20.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:20.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:20.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:20.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:20.063+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:20.146+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:20.146+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:20.146+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:20.147+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:21.050 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:20.736+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:21.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:20.736+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:21.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:20.736+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:21.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:20.737+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:21.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:21.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:21.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:21.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:21.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:21.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:21.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:21.050+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:21.174+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:21.174+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:21.174+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:21.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:21.175+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:22.019 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:21.707+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:22.019 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:21.707+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:22.019 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:21.707+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:22.020 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:21.708+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:22.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:22.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:22.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:22.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:22.018+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:22.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:22.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:22.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:22.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:22.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:22.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:22.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:22.221+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:22.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:28:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:28:22.555+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28718 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:28:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:22.746+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:22.746+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:22.746+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:22.747+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:23.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:23.220+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:23.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:23.220+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:23.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:23.220+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:23.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:23.221+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:23.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:23.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:23.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:23.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:23.059+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:24.024 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:23.788+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:24.024 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:23.788+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:24.024 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:23.788+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:24.025 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:23.789+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:24.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:24.199+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:24.199+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:24.199+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:24.200+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:24.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:24.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:24.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:24.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:24.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:24.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:24.022+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:25.023 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:24.744+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:25.023 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:24.744+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:25.023 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:24.744+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:25.023 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:24.745+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:25.024 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:25.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:25.024 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:25.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:25.026 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:25.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:25.026 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:25.021+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:25.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:25.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:25.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:25.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:25.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:25.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:25.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:25.240+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:26.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:26.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:26.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:26.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:26.038+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:26.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:25.792+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:26.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:25.792+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:26.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:25.792+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:26.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:25.792+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:26.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:26.233+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:26.233+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:26.233+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:26.234+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:27.054 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:26.752+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:27.055 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:26.752+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:27.055 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:26.752+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:27.055 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:26.752+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:27.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:27.052+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:27.052+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:27.052+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:27.052+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:27.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:27.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:27.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:27.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:27.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:27.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:27.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:27.271+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:27.768 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:28:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:28:27.557+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28732 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:28:28.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:27.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:27.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:27.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:28.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:27.766+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:28.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:28.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:28.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:28.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:28.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:28.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:28.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:28.090+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:28.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:28.255+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:28.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:28.255+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:28.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:28.255+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:28.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:28.255+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:29.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:28.789+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:28.789+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:28.789+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:28.790+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:29.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:29.244+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:29.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:29.244+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:29.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:29.244+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:29.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:29.245+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:29.101+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:29.101+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:29.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:29.101+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:29.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:29.102+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:30.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:29.787+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:29.787+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:29.787+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:29.788+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:30.390 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:30.107+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:30.390 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:30.107+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:30.390 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:30.107+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:30.390 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:30.108+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:30.391 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:30.231+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:30.391 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:30.231+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:30.391 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:30.231+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:30.391 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:30.232+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:31.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:30.834+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:30.834+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:30.834+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:30.834+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:31.389 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:31.093+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:31.390 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:31.093+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:31.390 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:31.093+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:31.390 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:31.094+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:31.390 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:31.244+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:31.391 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:31.244+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:31.391 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:31.244+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:31.391 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:31.245+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:31.815 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:31.814+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:31.815 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:31.814+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:31.815 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:31.814+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:32.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:31.814+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:32.052+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:32.052+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:32.052+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:32.052+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:32.563 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:32.256+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:32.563 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:32.256+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:32.563 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:32.256+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:32.563 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:32.256+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:32.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:28:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:28:32.561+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28745 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:28:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:32.771+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:32.771+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:32.771+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:32.772+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:33.302 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:33.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:33.302 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:33.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:33.302 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:33.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:33.302 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:33.039+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:33.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:33.300+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:33.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:33.300+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:33.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:33.300+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:33.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:33.301+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:34.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:33.790+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:33.790+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:33.790+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:33.790+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:34.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:34.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:34.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:34.084+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:34.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:34.303+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:34.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:34.303+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:34.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:34.303+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:34.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:34.304+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:35.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:34.764+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:35.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:34.764+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:35.079 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:34.764+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:35.079 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:34.765+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:35.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:35.076+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:35.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:35.076+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:35.076+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:35.077+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:35.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:35.318+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:35.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:35.318+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:35.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:35.318+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:35.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:35.319+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:36.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:35.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:36.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:35.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:36.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:35.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:36.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:35.754+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:36.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:36.319+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:36.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:36.319+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:36.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:36.319+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:36.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:36.320+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:36.078+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:36.078+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:36.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:36.078+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:36.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:36.079+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:37.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:36.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:36.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:36.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:36.767+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:37.364 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:37.081+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:37.364 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:37.081+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:37.364 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:37.081+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:37.365 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:37.081+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:37.772 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:37.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:37.772 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:37.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:37.772 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:37.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:37.772 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:37.363+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:37.773 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:28:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:28:37.564+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28759 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:28:38.040 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:37.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:38.040 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:37.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:38.041 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:37.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:38.041 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:37.771+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:38.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:38.039+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:38.039+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:38.039+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:38.039+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:38.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:38.391+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:38.757 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:38.391+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:38.757 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:38.391+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:38.757 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:38.392+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:39.030 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:38.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:39.031 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:38.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:39.031 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:38.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:39.031 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:38.755+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:39.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:39.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:39.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:39.029+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:39.742 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:39.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:39.743 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:39.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:39.743 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:39.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:39.743 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:39.385+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:40.004 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:39.740+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:40.004 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:39.741+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:40.004 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:39.741+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:40.005 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:39.741+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:40.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:40.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:40.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:40.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:40.003+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:40.694 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:40.340+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:40.695 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:40.340+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:40.695 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:40.340+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:40.695 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:40.341+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:40.962 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:40.692+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:40.962 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:40.692+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:40.962 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:40.692+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:40.963 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:40.693+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:41.276 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:40.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:41.277 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:40.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:41.277 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:40.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:41.277 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:40.961+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:41.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:41.327+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:41.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:41.327+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:41.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:41.327+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:41.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:41.328+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:41.856 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:41.673+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:41.856 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:41.673+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:41.856 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:41.673+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:41.857 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:41.673+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:42.192 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:41.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:42.193 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:41.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:42.193 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:41.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:42.193 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:41.954+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:42.568 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:42.376+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:42.568 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:42.376+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:42.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:42.376+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:42.569 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:42.377+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:42.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:28:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:28:42.567+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28773 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:28:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:42.650+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:42.650+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:42.650+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:42.651+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:43.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:43.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:43.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:43.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:43.003+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:43.606 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:43.339+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:43.606 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:43.339+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:43.606 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:43.339+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:43.606 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:43.340+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:43.989 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:43.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:43.990 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:43.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:43.990 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:43.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:43.990 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:43.605+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:44.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:43.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:43.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:43.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:43.989+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:44.641 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:44.601+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:44.641 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:44.601+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:44.641 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:44.601+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:44.641 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:44.602+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:44.641 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:44.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:44.642 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:44.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:44.642 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:44.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:44.642 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:44.363+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:45.028+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:45.028+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:45.028+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:45.029+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:45.604 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:45.360+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:45.605 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:45.360+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:45.605 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:45.360+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:45.605 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:45.360+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:46.041 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:45.602+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:46.041 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:45.602+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:46.042 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:45.602+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:46.042 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:45.603+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:46.323 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:46.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:46.323 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:46.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:46.323 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:46.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:46.323 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:46.040+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:46.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:46.321+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:46.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:46.321+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:46.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:46.321+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:46.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:46.322+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:47.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:46.650+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:47.076 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:46.650+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:47.077 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:46.650+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:47.077 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:46.650+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:47.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:47.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:47.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:47.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:47.075+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:47.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:47.307+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:47.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:47.307+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:47.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:47.307+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:47.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:47.307+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:47.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:28:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:28:47.569+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28787 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:28:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:47.653+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:47.653+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:47.653+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:47.654+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:48.069+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:48.069+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:48.069+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:48.070+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:48.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:48.278+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:48.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:48.278+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:48.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:48.278+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:48.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:48.279+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:49.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:48.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:48.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:48.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:48.617+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:49.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:49.099+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:49.099+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:49.099+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:49.099+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:49.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:49.235+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:49.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:49.235+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:49.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:49.235+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:49.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:49.236+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:50.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:49.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:50.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:49.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:50.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:49.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:50.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:49.619+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:50.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:50.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:50.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:50.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:50.125+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:50.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:50.250+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:50.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:50.250+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:50.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:50.250+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:50.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:50.250+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:51.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:50.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:51.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:50.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:51.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:50.604+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:51.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:50.605+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:51.448 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:51.164+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:51.449 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:51.164+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:51.449 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:51.164+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:51.449 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:51.164+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:51.449 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:51.244+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:51.450 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:51.244+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:51.450 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:51.244+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:51.450 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:51.245+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:51.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:51.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:51.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:51.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:51.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:51.589+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:52.574 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:52.240+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:52.575 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:52.240+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:52.575 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:52.240+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:52.575 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:52.241+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:52.575 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:52.145+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:52.575 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:52.145+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:52.575 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:52.145+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:52.576 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:52.146+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:52.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:28:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:28:52.572+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28800 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:28:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:52.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:52.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:52.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:52.573+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:53.535 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:53.190+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:53.535 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:53.190+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:53.535 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:53.190+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:53.535 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:53.191+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:53.536 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:53.210+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:53.536 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:53.210+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:53.536 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:53.210+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:53.536 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:53.211+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:53.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:53.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:53.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:53.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:53.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:53.534+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:54.534 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:54.183+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:54.534 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:54.183+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:54.535 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:54.183+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:54.535 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:54.184+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:54.535 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:54.255+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:54.535 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:54.255+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:54.535 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:54.255+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:54.535 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:54.256+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:54.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:54.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:54.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:54.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:54.533+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:55.537 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:55.160+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:55.538 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:55.160+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:55.538 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:55.160+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:55.538 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:55.161+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:55.538 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:55.257+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:55.538 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:55.257+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:55.539 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:55.257+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:55.539 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:55.258+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:55.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:55.536+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:55.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:55.536+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:55.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:55.536+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:55.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:55.537+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:56.538 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:56.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:56.539 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:56.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:56.539 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:56.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:56.539 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:56.169+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:56.539 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:56.298+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:56.539 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:56.298+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:56.539 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:56.298+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:56.540 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:56.298+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:56.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:56.537+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:56.537+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:56.537+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:56.538+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:57.548 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:57.180+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:57.548 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:57.180+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:57.549 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:57.180+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:57.549 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:57.180+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:57.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:57.281+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:57.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:57.281+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:57.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:57.281+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:57.549 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:57.282+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:57.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:28:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:28:57.575+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28814 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:28:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:57.547+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:57.547+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:57.547+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:57.547+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:58.549 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:58.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:58.549 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:58.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:58.549 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:58.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:58.549 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:58.230+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:58.550 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:58.301+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:58.550 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:58.301+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:58.550 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:58.301+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:58.550 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:58.302+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:58.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:58.547+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:58.547+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:58.547+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:58.548+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:59.506 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:59.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:59.506 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:59.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:59.507 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:59.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:28:59.507 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:28:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:28:59.187+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:28:59.507 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:59.300+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:59.507 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:59.300+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:59.507 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:59.300+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:28:59.507 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:28:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:28:59.301+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:28:59.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:59.504+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:59.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:59.504+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:59.504+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:28:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:28:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:28:59.506+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:00.546 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:00.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:00.546 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:00.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:00.547 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:00.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:00.547 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:00.230+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:00.547 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:00.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:00.547 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:00.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:00.547 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:00.330+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:00.547 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:00.331+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:00.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:00.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:00.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:00.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:00.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:00.543+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:01.552 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:01.231+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:01.553 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:01.231+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:01.553 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:01.231+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:01.553 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:01.232+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:01.553 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:01.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:01.553 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:01.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:01.553 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:01.313+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:01.554 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:01.314+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:01.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:01.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:01.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:01.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:01.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:01.550+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:02.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:02.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:02.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:02.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:02.341+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:02.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:02.342+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:02.269+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:02.269+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:02.269+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:02.270+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:03.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:29:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:29:02.577+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28828 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:29:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:02.599+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:02.599+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:02.599+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:03.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:02.600+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:03.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:03.300+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:03.300+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:03.300+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:03.301+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:03.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:03.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:03.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:03.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:03.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:03.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:03.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:03.379+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:04.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:03.641+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:03.641+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:03.641+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:03.642+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:04.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:04.411+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:04.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:04.411+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:04.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:04.411+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:04.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:04.412+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:04.280+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:04.280+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:04.280+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:04.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:04.280+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:05.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:04.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:05.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:04.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:05.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:04.638+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:05.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:04.639+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:05.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:05.247+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:05.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:05.247+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:05.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:05.247+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:05.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:05.248+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:05.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:05.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:05.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:05.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:05.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:05.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:05.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:05.446+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:05.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:05.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:05.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:05.593+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:06.475 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:06.250+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:06.475 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:06.250+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:06.475 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:06.250+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:06.476 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:06.251+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:06.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:06.473+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:06.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:06.473+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:06.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:06.473+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:06.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:06.474+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:06.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:06.629+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:06.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:06.629+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:06.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:06.629+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:06.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:06.630+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:07.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:07.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:07.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:07.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:07.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:07.496+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:07.293+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:07.293+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:07.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:07.293+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:07.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:07.293+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:08.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:07.623+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:07.623+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:07.623+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:07.624+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:08.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:29:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:29:07.581+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28842 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:29:08.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:08.311+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:08.311+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:08.311+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:08.312+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:08.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:08.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:08.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:08.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:08.542+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:08.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:08.543+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:09.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:08.602+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:09.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:08.602+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:09.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:08.602+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:09.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:08.602+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:09.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:09.301+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:09.301+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:09.301+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:09.301+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:09.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:09.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:09.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:09.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:09.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:09.496+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:09.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:09.496+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:10.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:09.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:10.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:09.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:10.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:09.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:10.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:09.631+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:10.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:10.323+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:10.323+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:10.323+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:10.324+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:10.584 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:10.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:10.584 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:10.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:10.584 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:10.483+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:10.584 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:10.484+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:11.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:10.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:11.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:10.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:11.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:10.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:11.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:10.589+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:11.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:11.303+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:11.303+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:11.303+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:11.304+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:11.469+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:11.469+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:11.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:11.469+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:11.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:11.470+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:11.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:11.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:11.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:11.631+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:12.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:12.302+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:12.302+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:12.302+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:12.302+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:12.433+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:12.433+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:12.433+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:12.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:12.433+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:13.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:29:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:29:12.584+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28855 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:29:13.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:12.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:13.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:12.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:13.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:12.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:13.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:12.625+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:13.626 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:13.344+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:13.626 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:13.344+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:13.626 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:13.344+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:13.626 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:13.345+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:13.627 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:13.421+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:13.627 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:13.421+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:13.627 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:13.421+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:13.627 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:13.422+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:14.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:13.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:13.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:13.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:13.625+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:14.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:14.393+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:14.393+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:14.393+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:14.394+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:14.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:14.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:14.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:14.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:14.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:14.579+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:14.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:14.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:14.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:14.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:14.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:14.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:14.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:14.471+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:15.732 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:15.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:15.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:15.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:15.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:15.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:15.733 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:15.460+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:15.733 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:15.391+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:15.733 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:15.391+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:15.734 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:15.391+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:15.734 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:15.392+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:15.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:15.558+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:15.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:15.558+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:15.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:15.558+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:15.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:15.559+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:16.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:16.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:16.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:16.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:16.593+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:16.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:16.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:16.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:16.422+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:16.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:16.423+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:16.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:16.414+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:16.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:16.414+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:16.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:16.414+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:16.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:16.414+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:17.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:29:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:29:17.587+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28869 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:29:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:17.452+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:17.452+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:17.452+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:17.453+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:17.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:17.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:17.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:17.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:17.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:17.378+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:17.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:17.379+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:17.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:17.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:17.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:17.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:17.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:17.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:17.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:17.589+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:18.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:18.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:18.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:18.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:18.494+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:18.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:18.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:18.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:18.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:18.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:18.590+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:18.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:18.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:18.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:18.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:18.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:18.362+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:18.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:18.362+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:19.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:19.447+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:19.447+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:19.447+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:19.448+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:19.598+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:19.598+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:19.598+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:19.599+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:19.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:19.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:19.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:19.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:19.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:19.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:19.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:19.399+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:20.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:20.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:20.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:20.410+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:20.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:20.376+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:20.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:20.376+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:20.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:20.376+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:20.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:20.376+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:20.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:20.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:20.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:20.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:20.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:20.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:20.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:20.563+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:21.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:21.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:21.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:21.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:21.403+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:21.606+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:21.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:21.606+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:21.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:21.606+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:21.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:21.607+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:21.376+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:21.376+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:21.376+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:21.377+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:22.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:29:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:29:22.589+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28883 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:29:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:22.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:22.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:22.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:22.410+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:22.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:22.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:22.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:22.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:22.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:22.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:22.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:22.574+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:22.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:22.414+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:22.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:22.414+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:22.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:22.414+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:22.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:22.414+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:23.702 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:23.441+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:23.702 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:23.441+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:23.703 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:23.441+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:23.703 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:23.441+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:23.703 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:23.583+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:23.703 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:23.583+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:23.703 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:23.583+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:23.703 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:23.584+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:23.704 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:23.371+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:23.704 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:23.371+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:23.704 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:23.371+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:23.704 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:23.372+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:24.558+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:24.558+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:24.558+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:24.559+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:24.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:24.422+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:24.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:24.422+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:24.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:24.422+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:24.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:24.423+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:24.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:24.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:24.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:24.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:24.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:24.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:24.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:24.417+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:25.438+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:25.438+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:25.438+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:25.438+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:25.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:25.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:25.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:25.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:25.451+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:25.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:25.452+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:25.603+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:25.603+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:25.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:25.603+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:25.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:25.604+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:26.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:26.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:26.413+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:26.413+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:26.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:26.412+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:26.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:26.412+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:26.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:26.412+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:26.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:26.413+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:26.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:26.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:26.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:26.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:26.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:26.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:26.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:26.631+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:27.459 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:27.379+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:27.459 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:27.379+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:27.459 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:27.379+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:27.460 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:27.379+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:27.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:27.457+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:27.457+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:27.457+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:27.457+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:27.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:29:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:29:27.592+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28897 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:29:27.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:27.648+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:27.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:27.648+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:27.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:27.648+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:27.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:27.649+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:28.674 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:28.494+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:28.674 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:28.494+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:28.674 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:28.494+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:28.674 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:28.495+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:28.675 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:28.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:28.675 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:28.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:28.675 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:28.403+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:28.675 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:28.404+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:29.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:28.672+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:28.672+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:28.672+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:28.673+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:29.703 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:29.382+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:29.703 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:29.382+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:29.703 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:29.382+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:29.704 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:29.383+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:29.704 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:29.510+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:29.704 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:29.510+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:29.704 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:29.510+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:29.704 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:29.511+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:30.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:29.701+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:29.701+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:29.701+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:29.702+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:30.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:30.656+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:30.656+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:30.656+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:30.657+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:30.423+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:30.423+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:30.423+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:30.424+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:30.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:30.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:30.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:30.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:30.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:30.487+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:30.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:30.488+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:31.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:31.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:31.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:31.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:31.532+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:31.617+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:31.617+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:31.617+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:31.618+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:31.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:31.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:31.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:31.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:31.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:31.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:31.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:31.464+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:32.508 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:32.430+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:32.508 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:32.430+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:32.509 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:32.430+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:32.509 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:32.431+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:32.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:32.506+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:32.506+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:32.506+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:32.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:32.507+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:32.645+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:32.645+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:32.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:32.645+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:32.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:32.646+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:32.832 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:29:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:29:32.595+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28910 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:29:33.793 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:33.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:33.793 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:33.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:33.793 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:33.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:33.793 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:33.462+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:33.793 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:33.507+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:33.794 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:33.507+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:33.794 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:33.507+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:33.794 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:33.508+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:33.794 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:33.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:33.794 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:33.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:33.795 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:33.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:33.795 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:33.625+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:34.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:34.494+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:34.494+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:34.494+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:34.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:34.495+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:34.649+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:34.649+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:34.649+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:34.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:34.650+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:34.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:34.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:34.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:34.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:34.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:34.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:34.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:34.508+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:35.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:35.478+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:35.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:35.479+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:35.479+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:35.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:35.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:35.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:35.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:35.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:35.619+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:35.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:35.471+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:35.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:35.471+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:35.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:35.471+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:35.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:35.471+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:36.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:36.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:36.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:36.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:36.496+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:36.509+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:36.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:36.509+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:36.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:36.509+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:36.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:36.510+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:36.575+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:36.575+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:36.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:36.575+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:36.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:36.575+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:37.508+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:37.508+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:37.508+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:37.509+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:37.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:37.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:37.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:37.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:37.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:37.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:37.569+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:37.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:37.451+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:37.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:37.451+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:37.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:37.451+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:37.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:37.452+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:37.833 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:29:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:29:37.597+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28924 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:29:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:38.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:38.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:38.548+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:38.549+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:38.566+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:38.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:38.566+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:38.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:38.566+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:38.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:38.567+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:38.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:38.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:38.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:38.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:38.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:38.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:38.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:38.500+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:39.595+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:39.595+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:39.595+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:39.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:39.596+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:39.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:39.457+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:39.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:39.457+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:39.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:39.457+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:39.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:39.457+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:39.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:39.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:39.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:39.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:39.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:39.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:39.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:39.556+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:40.595+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:40.595+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:40.595+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:40.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:40.596+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:40.530+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:40.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:40.530+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:40.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:40.530+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:40.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:40.531+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:40.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:40.477+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:40.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:40.477+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:40.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:40.477+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:40.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:40.478+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:41.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:41.571+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:41.571+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:41.571+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:41.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:41.572+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:41.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:41.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:41.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:41.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:41.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:41.534+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:41.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:41.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:41.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:41.503+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:42.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:29:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:29:42.600+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28938 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:29:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:42.578+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:42.578+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:42.578+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:42.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:42.579+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:42.558+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:42.558+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:42.558+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:42.559+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:42.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:42.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:42.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:42.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:42.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:42.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:42.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:42.478+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:43.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:43.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:43.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:43.574+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:43.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:43.575+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:43.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:43.551+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:43.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:43.551+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:43.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:43.551+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:43.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:43.552+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:43.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:43.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:43.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:43.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:43.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:43.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:43.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:43.517+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:44.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:44.599+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:44.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:44.599+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:44.834 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:44.599+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:44.834 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:44.599+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:44.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:44.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:44.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:44.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:44.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:44.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:44.835 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:44.512+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:44.835 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:44.498+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:44.835 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:44.498+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:44.835 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:44.498+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:44.835 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:44.499+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:45.731 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:45.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:45.732 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:45.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:45.732 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:45.476+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:45.732 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:45.477+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:45.732 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:45.620+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:45.733 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:45.620+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:45.733 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:45.620+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:45.733 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:45.621+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:45.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:45.485+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:45.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:45.485+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:45.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:45.485+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:45.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:45.485+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:46.524 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:46.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:46.524 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:46.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:46.524 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:46.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:46.525 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:46.486+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:46.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:46.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:46.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:46.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:46.651+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:46.522+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:46.522+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:46.522+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:46.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:46.523+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:47.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:29:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:29:47.604+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28952 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:29:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:47.647+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:47.647+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:47.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:47.647+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:47.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:47.647+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:47.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:47.520+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:47.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:47.520+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:47.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:47.520+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:47.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:47.521+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:47.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:47.476+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:47.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:47.476+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:47.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:47.476+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:47.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:47.477+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:48.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:48.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:48.627+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:48.628+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:48.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:48.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:48.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:48.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:48.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:48.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:48.507+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:48.529+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:48.529+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:48.529+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:48.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:48.529+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:49.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:49.501+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:49.501+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:49.501+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:49.501+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:49.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:49.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:49.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:49.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:49.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:49.487+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:49.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:49.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:49.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:49.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:49.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:49.670+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:49.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:49.671+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:50.649+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:50.649+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:50.649+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:50.650+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:50.528+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:50.528+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:50.528+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:50.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:50.529+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:50.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:50.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:50.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:50.491+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:51.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:51.662+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:51.662+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:51.662+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:51.663+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:51.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:51.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:51.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:51.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:51.449+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:51.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:51.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:51.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:51.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:51.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:51.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:51.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:51.548+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:52.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:52.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:52.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:52.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:52.588+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:52.630+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:52.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:52.630+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:52.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:52.630+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:52.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:52.630+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:52.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:52.459+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:52.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:52.459+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:52.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:52.459+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:52.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:52.460+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:52.833 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:29:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:29:52.607+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28965 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:29:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:53.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:53.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:53.584+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:53.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:53.585+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:53.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:53.603+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:53.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:53.603+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:53.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:53.603+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:53.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:53.603+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:53.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:53.452+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:53.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:53.452+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:53.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:53.452+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:53.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:53.452+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:54.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:54.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:54.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:54.548+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:54.549+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:54.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:54.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:54.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:54.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:54.610+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:54.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:54.610+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:54.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:54.461+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:54.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:54.461+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:54.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:54.461+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:54.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:54.462+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:55.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:55.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:55.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:55.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:55.557+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:55.583+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:55.583+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:55.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:55.583+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:55.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:55.584+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:55.466+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:55.466+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:55.466+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:55.467+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:56.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:56.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:56.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:56.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:56.516+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:56.542+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:56.542+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:56.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:56.542+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:56.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:56.543+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:56.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:56.491+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:56.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:56.491+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:56.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:56.491+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:56.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:56.491+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:57.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:29:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:29:57.609+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28979 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:29:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:57.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:57.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:57.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:57.516+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:57.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:57.539+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:57.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:57.539+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:57.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:57.539+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:57.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:57.540+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:57.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:57.530+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:57.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:57.530+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:57.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:57.530+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:57.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:57.531+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:58.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:58.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:58.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:58.547+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:58.548+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:58.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:58.539+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:58.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:58.539+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:58.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:58.539+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:58.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:58.539+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:58.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:58.498+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:58.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:58.498+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:58.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:58.498+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:58.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:58.499+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:59.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:59.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:59.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:59.531+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:29:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:29:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:29:59.532+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:29:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:59.530+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:59.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:59.530+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:59.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:59.530+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:29:59.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:29:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:29:59.531+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:29:59.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:59.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:59.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:59.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:59.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:59.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:29:59.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:29:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:29:59.511+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:00.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:00.574+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:00.574+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:00.574+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:00.574+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:00.499+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:00.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:00.499+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:00.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:00.499+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:00.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:00.500+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:00.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:00.580+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:00.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:00.580+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:00.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:00.580+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:00.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:00.581+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:01.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:01.600+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:01.600+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:01.600+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:01.600+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:01.483+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:01.483+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:01.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:01.483+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:01.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:01.483+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:01.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:01.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:01.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:01.555+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:02.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:30:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:30:02.612+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 28993 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:30:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:02.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:02.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:02.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:02.517+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:02.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:02.603+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:02.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:02.603+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:02.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:02.603+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:02.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:02.604+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:02.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:02.510+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:02.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:02.510+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:02.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:02.510+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:02.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:02.511+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:03.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:03.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:03.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:03.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:03.617+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:03.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:03.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:03.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:03.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:03.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:03.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:03.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:03.521+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:03.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:03.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:03.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:03.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:03.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:03.553+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:03.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:03.554+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:04.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:04.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:04.616+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:04.617+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:04.541+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:04.541+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:04.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:04.541+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:04.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:04.541+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:04.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:04.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:04.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:04.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:04.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:04.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:04.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:04.517+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:05.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:05.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:05.655+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:05.656+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:05.540+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:05.540+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:05.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:05.540+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:05.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:05.541+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:05.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:05.514+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:05.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:05.514+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:05.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:05.514+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:05.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:05.514+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:06.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:06.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:06.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:06.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:06.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:06.667+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:06.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:06.668+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:06.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:06.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:06.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:06.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:06.573+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:06.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:06.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:06.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:06.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:06.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:06.486+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:06.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:06.487+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:07.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:30:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:30:07.615+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29007 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:30:07.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:07.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:07.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:07.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:07.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:07.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:07.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:07.587+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:07.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:07.636+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:07.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:07.636+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:07.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:07.636+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:07.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:07.637+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:07.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:07.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:07.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:07.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:07.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:07.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:07.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:07.455+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:08.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:08.645+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:08.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:08.645+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:08.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:08.645+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:08.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:08.646+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:08.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:08.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:08.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:08.630+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:08.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:08.630+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:08.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:08.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:08.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:08.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:08.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:08.461+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:08.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:08.461+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:09.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:09.619+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:09.619+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:09.619+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:09.620+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:09.682+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:09.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:09.682+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:09.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:09.682+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:09.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:09.683+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:09.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:09.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:09.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:09.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:09.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:09.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:09.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:09.498+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:10.460+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:10.460+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:10.460+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:10.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:10.461+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:10.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:10.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:10.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:10.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:10.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:10.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:10.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:10.703+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:10.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:10.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:10.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:10.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:10.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:10.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:10.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:10.589+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:11.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:11.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:11.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:11.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:11.508+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:11.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:11.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:11.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:11.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:11.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:11.745+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:11.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:11.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:11.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:11.544+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:12.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:30:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:30:12.618+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29020 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:30:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:12.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:12.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:12.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:12.730+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:12.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:12.548+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:12.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:12.548+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:12.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:12.548+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:12.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:12.548+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:12.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:12.501+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:12.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:12.501+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:12.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:12.501+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:12.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:12.502+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:13.770 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:13.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:13.771 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:13.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:13.771 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:13.517+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:13.771 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:13.518+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:13.771 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:13.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:13.771 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:13.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:13.771 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:13.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:13.772 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:13.574+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:14.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:13.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:13.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:13.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:13.770+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:14.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:14.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:14.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:14.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:14.723 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:14.528+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:14.723 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:14.529+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:14.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:14.469+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:14.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:14.469+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:14.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:14.469+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:14.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:14.469+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:14.719+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:14.719+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:14.719+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:14.720+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:15.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:15.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:15.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:15.726+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:15.726+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:15.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:15.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:15.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:15.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:15.549+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:15.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:15.550+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:15.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:15.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:15.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:15.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:15.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:15.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:15.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:15.501+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:16.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:16.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:16.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:16.460+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:16.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:16.700+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:16.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:16.700+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:16.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:16.700+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:16.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:16.701+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:16.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:16.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:16.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:16.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:16.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:16.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:16.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:16.582+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:17.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:30:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:30:17.620+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29034 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:30:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:17.487+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:17.487+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:17.487+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:17.488+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:17.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:17.685+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:17.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:17.685+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:17.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:17.685+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:17.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:17.685+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:17.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:17.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:17.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:17.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:17.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:17.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:17.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:17.589+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:18.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:18.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:18.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:18.680+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:18.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:18.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:18.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:18.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:18.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:18.463+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:18.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:18.464+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:18.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:18.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:18.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:18.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:18.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:18.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:18.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:18.557+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:19.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:19.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:19.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:19.483+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:19.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:19.728+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:19.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:19.728+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:19.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:19.728+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:19.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:19.729+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:19.542+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:19.542+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:19.542+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:19.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:19.543+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:20.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:20.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:20.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:20.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:20.522+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:20.699+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:20.699+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:20.699+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:20.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:20.700+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:20.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:20.583+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:20.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:20.583+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:20.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:20.583+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:20.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:20.585+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:21.733 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:21.560+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:21.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:21.560+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:21.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:21.560+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:21.734 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:21.560+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:21.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:21.477+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:21.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:21.477+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:21.734 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:21.477+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:21.735 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:21.478+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:22.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:21.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:21.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:21.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:21.732+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:22.727 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:30:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:30:22.624+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29048 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:30:22.728 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:22.429+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:22.728 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:22.429+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:22.728 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:22.429+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:22.728 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:22.430+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:22.728 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:22.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:22.729 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:22.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:22.729 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:22.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:22.729 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:22.558+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:22.725+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:22.725+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:22.725+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:22.726+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:23.566 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:23.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:23.566 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:23.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:23.566 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:23.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:23.566 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:23.419+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:23.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:23.693+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:23.693+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:23.693+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:23.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:23.694+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:23.564+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:23.564+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:23.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:23.564+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:23.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:23.565+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:24.684 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:24.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:24.685 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:24.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:24.685 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:24.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:24.685 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:24.406+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:24.685 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:24.406+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:24.685 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:24.406+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:24.686 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:24.406+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:24.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:24.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:24.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:24.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:24.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:24.589+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:24.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:24.590+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:25.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:24.683+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:25.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:25.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:25.435+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:25.435+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:25.662+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:25.662+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:25.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:25.662+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:25.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:25.663+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:25.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:25.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:25.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:25.574+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:26.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:26.430+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:26.430+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:26.430+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:26.430+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:27.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:26.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:26.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:26.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:26.680+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:26.585+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:26.585+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:27.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:26.585+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:27.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:26.585+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:27.730 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:30:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:30:27.627+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29062 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:30:27.731 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:27.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:27.731 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:27.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:27.731 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:27.437+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:27.731 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:27.438+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:27.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:27.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:27.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:27.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:27.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:27.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:27.732 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:27.588+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:27.728+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:27.728+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:27.728+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:27.729+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:28.728 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:28.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:28.729 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:28.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:28.729 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:28.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:28.729 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:28.543+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:28.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:28.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:28.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:28.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:28.729 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:28.445+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:28.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:28.445+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:29.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:28.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:28.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:28.727+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:28.727+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:29.748 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:29.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:29.748 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:29.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:29.748 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:29.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:29.748 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:29.411+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:29.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:29.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:29.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:29.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:29.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:29.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:29.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:29.588+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:29.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:29.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:29.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:29.747+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:30.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:30.538+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:30.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:30.538+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:30.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:30.538+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:30.778 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:30.539+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:30.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:30.425+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:30.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:30.425+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:30.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:30.425+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:30.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:30.426+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:30.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:30.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:30.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:30.776+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:31.751 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:31.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:31.751 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:31.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:31.751 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:31.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:31.752 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:31.561+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:31.752 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:31.401+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:31.752 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:31.401+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:31.752 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:31.401+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:31.752 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:31.402+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:32.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:31.749+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:31.749+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:31.749+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:31.750+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:32.777 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:30:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:30:32.630+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29075 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:30:32.778 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:32.584+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:32.778 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:32.584+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:32.778 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:32.584+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:32.778 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:32.585+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:32.779 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:32.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:32.779 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:32.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:32.779 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:32.398+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:32.779 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:32.399+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:32.776+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:32.776+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:32.776+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:32.777+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:33.556 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:33.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:33.557 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:33.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:33.557 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:33.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:33.557 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:33.368+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:33.557 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:33.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:33.557 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:33.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:33.558 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:33.554+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:33.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:33.555+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:33.801+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:33.801+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:33.801+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:33.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:33.801+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:34.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:34.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:34.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:34.395+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:34.396+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:34.575+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:34.575+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:34.575+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:34.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:34.575+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:35.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:34.849+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:34.849+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:34.849+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:34.850+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:35.821 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:35.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:35.821 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:35.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:35.821 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:35.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:35.821 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:35.376+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:35.822 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:35.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:35.822 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:35.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:35.822 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:35.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:35.822 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:35.528+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:36.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:35.819+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:35.819+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:35.819+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:35.820+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:36.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:36.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:36.780 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:36.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:36.780 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:36.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:36.780 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:36.480+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:36.780 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:36.352+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:36.780 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:36.352+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:36.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:36.352+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:36.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:36.352+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:37.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:36.778+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:36.778+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:36.778+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:36.778+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:37.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:37.315+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:37.315+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:37.315+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:37.316+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:37.445+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:37.445+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:37.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:37.445+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:37.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:37.445+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:38.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:30:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:30:37.633+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29089 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:30:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:37.765+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:37.765+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:37.765+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:37.765+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:38.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:38.462+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:38.462+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:38.462+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:38.463+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:38.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:38.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:38.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:38.312+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:38.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:38.313+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:39.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:38.771+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:38.771+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:38.771+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:38.772+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:39.747 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:39.423+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:39.747 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:39.423+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:39.748 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:39.423+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:39.748 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:39.424+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:39.748 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:39.359+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:39.748 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:39.359+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:39.748 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:39.359+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:39.749 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:39.360+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:40.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:39.745+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:39.745+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:39.745+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:39.747+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:40.704 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:40.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:40.704 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:40.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:40.704 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:40.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:40.705 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:40.376+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:40.705 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:40.431+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:40.705 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:40.431+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:40.705 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:40.431+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:40.705 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:40.431+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:41.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:40.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:41.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:40.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:40.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:40.703+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:41.681 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:41.360+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:41.681 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:41.360+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:41.681 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:41.360+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:41.682 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:41.361+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:41.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:41.428+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:41.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:41.428+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:41.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:41.428+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:41.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:41.428+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:42.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:41.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:42.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:41.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:42.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:41.679+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:42.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:41.680+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:42.638 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:42.355+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:42.638 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:42.355+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:42.638 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:42.355+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:42.638 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:42.356+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:42.639 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:42.428+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:42.639 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:42.428+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:42.639 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:42.428+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:42.639 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:42.429+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:43.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:30:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:30:42.636+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29103 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:30:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:42.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:42.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:42.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:42.649+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:43.607 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:43.342+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:43.607 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:43.342+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:43.607 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:43.342+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:43.608 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:43.343+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:43.608 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:43.393+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:43.608 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:43.393+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:43.608 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:43.393+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:43.608 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:43.393+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:44.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:43.662+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:43.662+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:43.662+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:43.662+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:44.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:44.294+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:44.294+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:44.294+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:44.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:44.294+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:44.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:44.441+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:44.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:44.441+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:44.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:44.441+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:44.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:44.442+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:45.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:44.701+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:44.701+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:44.701+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:44.702+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:45.655 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:45.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:45.656 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:45.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:45.656 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:45.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:45.656 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:45.426+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:45.656 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:45.344+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:45.657 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:45.344+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:45.657 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:45.344+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:45.657 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:45.345+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:46.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:45.654+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:45.654+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:45.654+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:45.654+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:46.685 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:46.386+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:46.685 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:46.386+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:46.686 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:46.386+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:46.686 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:46.387+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:46.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:46.386+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:46.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:46.386+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:46.686 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:46.386+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:46.687 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:46.387+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:47.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:46.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:46.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:46.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:46.684+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:47.641 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:47.385+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:47.642 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:47.385+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:47.642 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:47.385+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:47.642 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:47.386+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:47.642 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:47.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:47.642 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:47.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:47.643 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:47.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:47.643 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:47.419+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:48.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:30:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:30:47.639+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29117 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:30:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:47.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:47.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:47.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:47.651+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:48.704 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:48.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:48.705 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:48.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:48.705 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:48.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:48.705 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:48.401+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:48.706 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:48.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:48.706 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:48.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:48.706 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:48.638+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:48.706 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:48.639+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:48.707 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:48.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:48.707 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:48.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:48.707 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:48.454+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:48.707 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:48.454+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:49.675 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:49.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:49.676 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:49.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:49.676 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:49.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:49.676 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:49.399+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:49.676 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:49.455+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:49.677 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:49.455+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:49.677 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:49.455+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:49.677 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:49.455+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:50.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:49.674+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:49.674+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:49.674+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:49.675+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:50.708 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:50.471+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:50.708 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:50.471+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:50.709 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:50.471+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:50.709 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:50.472+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:50.709 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:50.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:50.709 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:50.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:50.709 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:50.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:50.709 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:50.362+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:51.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:50.706+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:51.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:50.707+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:50.707+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:50.707+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:51.701 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:51.341+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:51.701 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:51.341+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:51.701 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:51.341+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:51.701 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:51.341+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:51.702 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:51.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:51.702 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:51.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:51.702 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:51.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:51.702 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:51.511+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:52.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:51.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:51.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:51.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:51.699+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:52.644 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:52.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:52.644 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:52.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:52.644 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:52.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:52.645 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:52.483+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:52.645 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:52.384+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:52.645 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:52.384+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:52.645 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:52.384+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:52.645 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:52.384+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:53.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:30:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:30:52.641+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29130 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:30:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:52.676+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:52.676+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:52.676+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:52.677+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:53.708 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:53.415+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:53.708 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:53.415+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:53.708 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:53.415+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:53.708 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:53.416+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:53.709 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:53.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:53.709 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:53.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:53.709 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:53.522+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:53.709 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:53.523+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:54.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:53.706+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:53.706+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:53.706+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:53.707+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:54.692 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:54.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:54.693 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:54.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:54.693 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:54.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:54.693 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:54.419+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:54.694 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:54.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:54.694 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:54.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:54.694 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:54.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:54.694 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:54.481+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:55.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:54.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:55.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:54.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:55.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:54.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:55.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:54.691+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:55.741 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:55.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:55.741 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:55.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:55.741 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:55.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:55.742 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:55.449+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:55.742 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:55.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:55.742 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:55.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:55.742 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:55.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:55.742 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:55.493+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:56.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:55.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:55.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:55.738+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:55.739+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:56.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:56.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:56.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:56.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:56.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:56.470+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:56.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:56.470+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:56.757 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:56.455+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:56.757 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:56.455+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:56.757 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:56.455+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:56.757 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:56.456+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:57.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:56.753+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:56.753+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:56.753+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:56.754+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:57.805 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:57.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:57.805 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:57.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:57.805 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:57.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:57.805 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:57.480+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:57.805 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:57.503+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:57.806 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:57.503+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:57.806 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:57.503+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:57.806 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:57.503+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:57.806 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:30:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:30:57.644+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29144 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:30:58.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:57.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:57.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:57.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:57.804+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:58.810 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:58.548+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:58.810 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:58.548+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:58.810 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:58.548+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:58.810 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:58.549+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:58.811 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:58.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:58.811 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:58.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:58.811 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:58.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:58.811 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:58.512+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:59.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:58.808+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:58.808+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:58.808+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:30:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:58.809+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:30:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:59.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:59.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:59.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:30:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:30:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:30:59.490+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:30:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:59.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:59.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:59.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:30:59.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:30:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:30:59.521+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:00.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:59.846+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:59.846+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:59.846+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:30:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:30:59.846+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:00.819 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:00.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:00.819 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:00.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:00.820 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:00.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:00.820 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:00.526+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:00.820 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:00.511+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:00.820 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:00.511+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:00.820 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:00.511+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:00.820 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:00.511+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:00.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:00.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:00.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:01.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:00.818+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:01.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:01.782+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:01.782+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:01.782+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:01.783+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:01.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:01.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:01.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:01.555+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:01.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:01.555+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:01.546+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:01.546+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:01.546+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:01.547+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:02.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:31:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:31:02.648+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29158 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:31:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:02.735+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:02.735+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:02.735+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:02.736+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:02.518+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:02.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:02.518+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:02.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:02.518+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:02.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:02.519+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:02.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:02.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:02.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:02.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:02.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:02.499+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:02.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:02.500+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:03.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:03.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:03.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:03.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:03.515+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:03.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:03.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:03.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:03.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:03.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:03.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:03.737+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:03.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:03.559+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:03.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:03.559+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:03.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:03.559+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:03.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:03.560+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:04.757 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:04.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:04.757 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:04.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:04.757 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:04.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:04.757 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:04.526+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:04.757 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:04.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:04.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:04.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:04.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:04.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:04.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:04.501+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:05.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:04.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:04.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:04.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:04.756+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:05.789 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:05.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:05.790 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:05.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:05.790 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:05.533+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:05.790 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:05.534+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:05.790 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:05.504+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:05.790 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:05.504+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:05.790 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:05.504+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:05.790 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:05.505+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:06.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:05.788+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:05.788+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:05.788+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:05.788+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:06.817 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:06.465+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:06.817 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:06.465+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:06.817 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:06.465+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:06.817 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:06.466+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:06.818 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:06.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:06.818 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:06.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:06.818 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:06.555+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:06.818 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:06.555+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:07.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:06.815+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:06.815+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:06.815+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:06.816+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:07.795 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:31:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:31:07.651+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29172 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:31:07.795 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:07.476+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:07.796 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:07.476+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:07.796 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:07.476+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:07.796 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:07.477+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:07.796 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:07.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:07.796 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:07.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:07.796 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:07.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:07.796 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:07.569+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:07.794+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:07.794+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:07.794+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:07.794+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:08.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:08.445+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:08.824 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:08.445+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:08.824 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:08.445+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:08.824 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:08.446+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:08.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:08.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:08.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:08.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:08.825 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:08.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:08.825 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:08.616+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:09.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:08.822+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:09.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:08.822+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:09.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:08.822+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:09.075 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:08.822+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:09.815 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:09.401+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:09.815 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:09.401+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:09.816 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:09.401+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:09.816 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:09.402+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:09.816 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:09.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:09.816 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:09.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:09.816 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:09.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:09.816 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:09.612+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:09.813+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:09.813+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:09.813+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:09.814+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:10.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:10.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:10.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:10.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:10.615+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:10.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:10.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:10.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:10.418+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:10.839+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:10.839+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:10.839+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:10.840+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:11.798 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:11.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:11.798 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:11.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:11.798 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:11.440+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:11.798 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:11.440+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:11.798 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:11.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:11.799 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:11.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:11.799 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:11.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:11.799 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:11.592+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:12.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:11.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:12.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:11.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:12.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:11.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:12.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:11.797+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:12.760 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:12.567+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:12.761 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:12.567+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:12.761 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:12.567+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:12.761 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:12.567+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:12.761 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:12.486+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:12.762 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:12.486+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:12.762 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:12.486+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:12.762 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:12.487+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:12.762 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:31:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:31:12.653+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29182 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:31:13.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:12.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:12.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:12.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:12.760+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:13.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:13.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:13.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:13.733+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:13.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:13.733+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:13.526+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:13.526+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:13.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:13.526+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:13.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:13.527+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:13.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:13.484+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:13.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:13.484+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:13.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:13.484+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:13.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:13.485+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:14.774 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:14.515+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:14.774 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:14.515+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:14.774 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:14.515+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:14.774 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:14.516+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:14.775 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:14.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:14.775 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:14.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:14.775 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:14.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:14.775 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:14.513+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:15.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:14.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:15.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:14.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:14.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:14.773+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:15.738 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:15.503+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:15.738 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:15.503+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:15.738 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:15.503+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:15.739 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:15.504+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:15.739 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:15.521+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:15.739 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:15.521+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:15.739 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:15.521+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:15.739 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:15.521+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:16.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:15.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:15.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:15.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:15.737+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:16.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:16.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:16.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:16.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:16.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:16.698+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:16.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:16.699+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:16.511+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:16.511+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:16.511+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:16.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:16.512+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:16.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:16.548+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:16.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:16.548+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:16.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:16.548+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:16.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:16.549+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:17.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:31:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:31:17.656+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29190 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:31:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:17.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:17.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:17.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:17.497+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:17.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:17.731+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:17.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:17.731+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:17.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:17.731+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:17.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:17.732+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:17.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:17.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:17.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:17.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:17.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:17.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:17.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:17.528+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:18.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:18.468+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:18.468+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:18.468+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:18.469+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:18.503+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:18.503+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:18.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:18.503+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:18.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:18.504+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:18.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:18.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:18.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:18.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:18.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:18.702+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:18.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:18.703+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:19.459+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:19.459+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:19.459+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:19.460+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:19.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:19.501+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:19.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:19.501+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:19.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:19.501+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:19.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:19.501+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:19.751+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:19.751+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:19.751+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:19.751+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:20.762 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:20.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:20.762 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:20.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:20.762 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:20.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:20.762 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:20.457+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:20.763 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:20.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:20.763 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:20.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:20.763 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:20.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:20.763 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:20.419+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:21.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:20.760+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:20.760+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:20.760+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:20.761+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:21.748 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:21.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:21.749 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:21.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:21.749 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:21.489+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:21.749 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:21.490+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:21.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:21.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:21.749 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:21.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:21.750 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:21.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:21.750 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:21.428+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:22.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:21.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:21.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:21.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:21.747+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:22.752 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:31:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:31:22.659+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29197 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:31:22.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:22.538+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:22.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:22.538+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:22.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:22.538+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:22.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:22.539+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:22.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:22.435+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:22.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:22.435+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:22.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:22.435+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:22.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:22.436+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:22.751+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:22.751+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:22.751+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:22.751+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:23.709 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:23.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:23.709 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:23.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:23.710 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:23.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:23.710 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:23.566+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:23.710 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:23.431+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:23.710 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:23.431+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:23.710 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:23.431+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:23.710 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:23.432+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:24.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:23.789+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:23.789+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:23.789+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:23.790+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:24.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:24.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:24.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:24.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:24.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:24.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:24.778 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:24.578+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:24.778 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:24.467+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:24.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:24.467+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:24.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:24.467+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:24.779 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:24.467+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:25.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:24.776+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:24.776+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:24.776+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:24.777+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:25.743 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:25.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:25.743 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:25.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:25.744 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:25.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:25.744 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:25.581+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:25.744 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:25.487+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:25.745 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:25.487+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:25.745 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:25.487+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:25.745 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:25.487+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:26.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:25.741+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:26.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:25.741+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:25.741+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:25.742+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:26.474+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:26.474+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:26.474+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:26.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:26.475+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:26.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:26.700+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:26.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:26.700+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:26.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:26.700+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:26.833 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:26.701+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:26.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:26.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:26.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:26.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:26.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:26.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:26.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:26.569+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:27.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:31:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:31:27.662+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29205 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:31:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:27.712+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:27.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:27.712+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:27.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:27.712+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:27.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:27.713+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:27.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:27.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:27.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:27.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:27.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:27.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:27.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:27.535+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:27.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:27.468+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:27.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:27.468+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:27.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:27.468+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:27.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:27.469+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:28.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:28.550+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:28.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:28.550+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:28.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:28.550+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:28.730 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:28.550+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:28.730 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:28.450+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:28.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:28.450+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:28.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:28.450+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:28.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:28.451+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:29.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:28.728+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:28.728+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:28.728+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:28.729+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:29.780 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:29.443+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:29.781 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:29.443+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:29.781 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:29.443+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:29.781 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:29.444+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:29.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:29.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:29.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:29.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:29.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:29.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:29.782 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:29.503+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:29.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:29.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:29.777+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:29.778+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:30.801 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:30.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:30.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:30.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:30.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:30.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:30.802 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:30.510+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:30.802 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:30.456+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:30.802 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:30.456+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:30.802 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:30.456+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:30.803 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:30.457+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:30.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:30.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:30.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:30.800+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:31.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:31.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:31.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:31.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:31.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:31.534+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:31.758 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:31.534+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:31.759 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:31.409+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:31.759 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:31.409+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:31.759 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:31.409+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:31.759 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:31.410+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:32.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:31.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:31.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:31.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:31.756+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:32.775 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:31:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:31:32.664+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29212 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:31:32.776 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:32.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:32.776 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:32.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:32.776 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:32.449+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:32.776 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:32.450+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:32.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:32.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:32.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:32.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:32.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:32.581+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:32.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:32.582+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:33.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:32.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:32.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:32.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:32.773+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:33.710 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:33.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:33.710 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:33.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:33.711 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:33.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:33.711 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:33.630+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:33.711 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:33.421+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:33.711 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:33.421+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:33.712 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:33.421+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:33.712 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:33.422+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:34.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:33.762+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:33.762+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:33.762+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:33.763+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:34.755 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:34.403+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:34.755 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:34.403+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:34.755 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:34.403+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:34.756 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:34.404+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:34.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:34.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:34.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:34.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:34.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:34.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:34.756 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:34.643+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:34.752+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:34.752+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:34.752+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:34.753+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:35.803 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:35.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:35.803 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:35.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:35.804 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:35.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:35.804 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:35.405+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:35.804 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:35.606+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:35.804 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:35.606+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:35.804 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:35.606+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:35.805 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:35.607+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:36.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:35.801+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:35.801+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:35.801+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:35.801+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:36.793 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:36.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:36.793 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:36.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:36.793 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:36.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:36.794 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:36.371+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:36.794 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:36.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:36.794 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:36.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:36.794 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:36.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:36.794 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:36.610+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:37.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:36.791+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:36.791+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:36.791+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:36.792+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:37.601 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:37.339+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:37.601 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:37.339+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:37.601 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:37.339+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:37.601 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:37.340+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:38.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:37.599+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:37.599+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:37.599+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:37.600+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:37.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:37.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:38.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:37.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:38.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:37.769+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:38.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:31:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:31:37.667+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29220 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:31:38.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:38.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:38.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:38.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:38.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:38.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:38.650 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:38.362+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:39.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:38.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:38.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:38.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:38.647+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:38.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:38.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:38.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:38.733+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:39.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:39.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:39.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:39.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:39.316+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:40.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:39.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:39.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:39.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:39.608+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:39.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:39.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:40.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:39.687+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:40.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:39.688+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:40.646 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:40.355+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:40.646 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:40.355+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:40.646 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:40.355+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:40.647 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:40.356+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:41.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:40.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:40.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:40.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:40.645+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:40.645+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:40.645+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:40.645+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:41.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:40.645+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:41.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:41.326+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:41.326+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:41.326+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:41.327+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:42.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:41.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:41.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:41.638+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:41.639+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:42.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:41.680+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:42.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:41.680+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:42.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:41.680+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:42.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:41.681+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:42.673 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:42.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:42.673 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:42.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:42.673 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:42.592+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:42.673 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:42.593+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:42.674 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:42.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:42.674 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:42.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:42.674 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:42.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:42.674 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:42.371+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:43.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:31:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:31:42.670+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29227 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:31:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:42.715+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:42.715+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:42.715+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:42.716+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:43.698 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:43.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:43.698 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:43.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:43.698 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:43.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:43.698 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:43.587+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:43.699 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:43.338+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:43.699 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:43.338+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:43.699 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:43.338+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:43.699 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:43.338+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:44.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:43.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:43.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:43.696+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:43.697+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:44.707 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:44.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:44.707 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:44.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:44.707 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:44.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:44.707 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:44.352+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:44.708 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:44.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:44.708 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:44.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:44.708 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:44.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:44.708 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:44.598+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:45.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:44.705+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:44.705+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:44.705+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:44.706+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:45.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:45.319+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:45.584 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:45.319+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:45.584 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:45.319+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:45.584 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:45.320+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:45.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:45.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:45.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:45.723+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:45.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:46.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:45.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:46.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:45.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:46.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:45.635+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:46.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:46.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:46.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:46.370+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:46.371+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:46.701+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:46.701+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:47.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:46.701+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:47.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:46.701+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:47.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:46.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:47.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:46.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:47.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:46.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:47.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:46.596+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:47.675 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:47.599+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:47.676 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:47.599+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:47.676 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:47.599+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:47.676 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:47.600+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:47.676 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:47.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:47.676 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:47.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:47.676 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:47.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:47.677 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:47.421+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:48.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:31:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:31:47.673+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29235 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:31:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:47.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:47.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:47.683+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:47.684+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:48.683 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:48.407+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:48.683 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:48.407+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:48.683 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:48.407+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:48.683 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:48.407+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:48.684 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:48.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:48.684 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:48.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:48.684 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:48.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:48.684 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:48.649+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:49.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:48.681+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:48.681+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:48.681+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:48.682+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:49.721 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:49.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:49.722 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:49.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:49.722 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:49.608+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:49.722 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:49.608+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:49.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:49.457+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:49.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:49.457+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:49.723 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:49.457+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:49.723 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:49.458+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:50.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:49.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:50.085 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:49.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:50.085 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:49.720+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:50.085 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:49.721+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:50.710 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:50.574+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:50.711 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:50.574+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:50.711 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:50.574+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:50.711 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:50.575+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:50.711 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:50.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:50.711 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:50.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:50.712 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:50.413+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:50.712 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:50.414+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:51.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:50.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:50.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:50.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:50.709+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:51.715 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:51.565+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:51.716 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:51.565+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:51.716 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:51.565+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:51.716 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:51.566+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:51.716 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:51.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:51.716 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:51.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:51.716 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:51.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:51.721 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:51.389+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:52.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:51.714+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:52.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:51.714+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:51.714+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:51.714+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:52.691 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:52.432+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:52.692 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:52.432+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:52.692 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:52.432+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:52.692 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:52.433+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:52.692 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:52.560+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:52.692 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:52.560+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:52.692 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:52.560+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:52.693 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:52.561+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:52.693 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:31:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:31:52.676+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29242 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:31:53.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:52.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:52.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:52.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:53.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:52.690+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:53.711 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:53.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:53.711 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:53.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:53.711 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:53.600+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:53.712 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:53.600+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:53.712 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:53.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:53.712 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:53.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:53.712 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:53.426+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:53.712 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:53.427+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:54.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:53.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:54.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:53.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:53.729+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:53.730+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:54.745 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:54.425+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:54.746 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:54.425+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:54.746 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:54.425+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:54.746 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:54.426+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:54.747 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:54.573+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:54.747 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:54.573+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:54.747 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:54.573+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:54.747 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:54.574+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:55.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:54.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:55.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:54.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:55.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:54.744+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:55.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:54.745+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:55.751 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:55.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:55.752 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:55.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:55.752 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:55.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:55.752 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:55.425+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:55.752 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:55.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:55.752 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:55.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:55.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:55.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:55.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:55.621+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:56.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:55.750+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:56.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:55.750+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:55.750+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:55.751+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:56.795 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:56.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:56.795 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:56.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:56.795 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:56.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:56.796 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:56.592+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:56.796 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:56.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:56.796 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:56.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:56.796 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:56.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:56.796 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:56.411+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:57.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:56.793+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:56.793+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:56.793+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:56.794+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:57.681 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:57.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:57.681 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:57.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:57.681 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:57.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:57.681 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:57.628+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:57.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:57.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:57.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:57.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:57.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:57.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:57.682 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:57.390+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:58.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:31:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:31:57.679+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29250 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:31:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:57.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:57.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:57.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:57.836+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:58.805 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:58.396+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:58.805 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:58.396+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:58.805 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:58.396+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:58.805 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:58.397+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:58.806 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:58.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:58.806 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:58.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:58.806 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:58.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:58.806 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:58.596+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:59.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:58.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:58.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:58.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:31:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:58.804+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:31:59.768 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:59.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:59.768 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:59.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:59.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:59.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:31:59.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:31:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:31:59.570+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:31:59.769 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:59.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:59.769 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:59.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:59.769 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:59.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:31:59.769 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:31:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:31:59.410+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:00.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:59.767+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:00.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:59.767+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:00.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:59.767+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:00.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:31:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:31:59.767+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:00.797 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:00.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:00.797 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:00.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:00.798 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:00.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:00.798 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:00.404+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:00.798 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:00.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:00.798 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:00.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:00.798 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:00.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:00.798 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:00.610+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:00.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:00.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:00.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:00.796+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:01.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:01.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:01.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:01.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:01.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:01.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:01.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:01.424+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:01.823 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:01.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:01.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:01.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:01.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:01.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:01.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:01.618+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:02.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:01.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:01.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:01.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:01.823+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:02.636 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:02.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:02.637 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:02.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:02.637 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:02.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:02.637 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:02.384+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:03.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:32:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:32:02.682+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29257 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:32:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:02.869+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:02.869+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:02.869+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:02.870+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:03.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:02.635+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:03.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:02.635+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:03.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:02.635+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:03.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:02.635+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:03.676 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:03.388+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:03.677 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:03.388+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:03.677 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:03.388+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:03.677 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:03.389+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:04.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:03.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:03.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:03.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:03.905+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:03.675+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:03.675+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:04.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:03.675+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:04.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:03.676+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:04.693 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:04.382+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:04.693 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:04.382+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:04.694 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:04.382+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:04.694 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:04.382+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:05.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:04.898+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:04.898+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:04.898+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:04.899+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:04.692+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:05.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:04.692+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:05.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:04.692+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:05.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:04.692+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:05.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:05.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:05.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:05.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:05.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:05.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:05.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:05.353+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:05.868+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:05.868+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:05.868+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:05.869+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:05.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:06.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:05.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:06.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:05.659+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:06.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:05.660+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:06.629 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:06.365+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:06.630 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:06.365+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:06.630 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:06.365+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:06.630 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:06.366+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:06.899 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:06.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:06.899 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:06.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:06.899 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:06.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:06.899 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:06.628+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:07.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:06.896+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:06.896+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:06.896+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:06.897+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:07.687 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:07.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:07.687 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:07.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:07.687 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:07.410+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:07.687 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:07.411+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:07.688 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:07.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:07.688 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:07.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:07.688 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:07.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:07.688 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:07.658+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:07.937 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:32:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:32:07.684+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29265 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:32:07.937 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:07.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:07.938 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:07.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:07.938 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:07.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:07.938 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:07.934+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:08.691 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:08.365+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:08.691 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:08.365+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:08.691 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:08.365+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:08.692 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:08.366+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:08.954 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:08.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:08.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:08.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:08.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:08.688+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:08.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:08.689+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:09.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:08.952+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:08.952+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:08.952+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:08.952+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:09.717 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:09.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:09.717 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:09.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:09.717 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:09.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:09.717 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:09.395+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:10.000 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:09.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:10.000 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:09.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:10.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:09.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:10.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:09.715+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:10.320 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:09.997+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:10.337 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:09.997+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:10.337 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:09.997+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:10.338 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:09.998+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:10.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:10.379+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:10.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:10.379+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:10.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:10.379+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:10.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:10.380+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:11.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:10.723+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:10.723+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:10.723+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:10.723+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:10.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:10.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:10.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:11.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:10.961+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:11.695 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:11.405+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:11.695 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:11.405+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:11.696 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:11.405+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:11.696 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:11.405+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:11.973 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:11.694+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:11.973 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:11.694+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:11.974 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:11.694+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:11.974 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:11.694+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:12.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:11.972+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:11.972+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:11.972+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:11.972+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:12.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:32:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:32:12.688+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29272 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:32:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:12.454+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:12.454+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:12.454+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:12.455+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:12.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:12.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:12.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:12.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:12.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:12.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:12.667+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:13.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:12.991+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:12.991+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:12.991+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:12.992+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:13.713 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:13.439+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:13.714 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:13.439+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:13.714 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:13.439+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:13.714 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:13.440+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:13.714 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:13.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:13.714 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:13.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:13.715 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:13.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:13.715 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:13.657+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:14.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:14.009+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:14.009+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:14.009+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:14.010+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:14.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:14.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:14.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:14.609+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:14.610+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:14.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:14.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:14.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:14.490+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:14.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:14.491+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:14.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:14.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:14.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:14.966+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:15.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:15.538+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:15.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:15.538+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:15.649 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:15.538+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:15.650 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:15.539+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:15.921 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:15.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:15.921 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:15.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:15.921 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:15.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:15.921 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:15.648+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:16.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:15.919+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:15.919+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:15.919+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:15.920+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:16.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:16.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:16.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:16.550+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:16.551+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:16.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:16.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:16.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:16.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:16.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:16.686+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:16.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:16.686+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:17.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:16.918+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:16.918+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:16.918+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:16.918+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:17.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:32:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:32:17.690+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29280 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:32:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:17.692+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:17.692+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:17.692+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:17.693+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:17.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:17.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:17.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:17.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:17.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:17.561+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:17.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:17.562+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:17.909+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:17.909+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:17.909+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:17.910+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:18.872 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:18.712+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:18.872 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:18.712+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:18.872 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:18.712+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:18.872 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:18.712+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:18.873 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:18.594+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:18.873 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:18.594+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:18.873 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:18.594+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:18.873 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:18.595+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:19.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:18.870+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:19.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:18.870+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:19.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:18.870+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:19.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:18.871+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:19.826 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:19.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:19.827 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:19.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:19.827 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:19.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:19.827 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:19.670+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:19.827 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:19.564+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:19.827 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:19.564+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:19.827 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:19.564+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:19.828 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:19.564+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:19.824+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:19.824+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:19.824+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:19.825+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:20.780 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:20.623+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:20.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:20.623+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:20.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:20.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:20.781 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:20.624+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:20.781 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:20.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:20.782 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:20.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:20.782 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:20.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:20.782 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:20.594+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:21.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:20.779+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:20.779+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:20.779+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:20.779+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:21.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:21.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:21.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:21.747+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:22.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:21.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:22.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:21.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:22.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:21.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:22.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:21.664+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:22.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:21.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:22.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:21.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:22.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:21.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:22.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:21.582+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:22.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:32:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:32:22.693+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29287 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:32:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:22.787+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:22.787+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:22.787+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:22.788+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:22.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:22.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:22.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:22.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:22.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:22.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:22.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:22.544+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:22.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:22.700+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:22.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:22.700+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:22.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:22.700+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:22.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:22.700+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:23.786 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:23.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:23.787 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:23.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:23.787 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:23.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:23.787 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:23.530+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:23.787 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:23.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:23.787 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:23.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:23.787 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:23.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:23.788 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:23.682+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:23.788 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:23.785+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:23.788 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:23.785+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:23.788 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:23.785+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:24.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:23.785+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:24.820 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:24.553+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:24.820 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:24.553+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:24.820 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:24.553+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:24.821 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:24.553+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:24.821 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:24.724+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:24.821 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:24.724+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:24.821 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:24.724+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:24.821 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:24.725+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:25.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:24.818+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:24.818+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:24.818+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:24.819+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:25.813+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:25.813+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:25.813+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:25.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:25.814+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:25.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:25.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:25.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:25.573+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:25.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:25.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:25.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:25.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:25.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:25.704+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:25.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:25.705+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:26.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:26.802+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:26.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:26.802+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:26.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:26.802+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:26.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:26.803+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:26.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:26.560+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:26.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:26.560+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:26.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:26.560+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:26.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:26.561+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:26.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:26.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:26.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:26.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:26.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:26.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:26.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:26.664+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:27.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:32:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:32:27.696+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29295 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:32:27.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:27.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:27.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:27.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:27.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:27.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:27.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:27.757+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:27.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:27.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:27.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:27.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:27.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:27.572+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:27.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:27.573+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:27.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:27.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:27.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:27.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:27.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:27.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:27.835 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:27.629+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:29.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:28.723+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:28.723+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:28.723+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:28.724+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:28.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:28.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:29.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:28.593+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:29.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:28.594+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:29.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:28.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:29.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:28.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:29.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:28.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:29.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:28.633+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:29.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:29.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:29.736+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:29.737+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:29.621+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:30.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:29.621+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:30.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:29.621+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:30.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:29.621+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:30.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:29.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:30.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:29.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:30.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:29.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:30.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:29.623+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:31.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:30.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:30.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:30.660+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:30.660+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:30.776+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:30.776+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:31.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:30.776+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:31.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:30.777+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:31.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:30.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:31.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:30.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:31.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:30.581+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:31.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:30.582+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:31.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:31.818+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:31.818+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:31.818+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:31.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:31.819+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:31.574+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:31.574+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:31.574+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:31.575+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:31.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:31.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:31.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:31.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:31.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:31.620+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:31.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:31.620+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:32.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:32:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:32:32.699+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29302 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:32:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:32.608+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:32.608+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:32.608+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:32.609+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:32.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:32.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:32.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:32.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:32.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:32.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:32.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:32.578+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:32.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:32.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:32.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:32.836+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:33.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:33.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:33.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:34.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:33.620+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:34.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:33.826+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:34.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:33.826+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:34.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:33.826+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:34.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:33.827+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:34.084 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:33.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:34.084 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:33.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:34.084 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:33.590+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:34.085 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:33.591+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:34.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:34.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:34.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:34.608+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:35.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:34.795+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:35.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:34.795+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:35.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:34.795+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:35.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:34.796+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:35.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:34.598+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:35.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:34.598+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:35.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:34.598+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:35.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:34.599+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:35.783 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:35.601+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:35.784 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:35.601+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:35.784 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:35.601+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:35.784 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:35.602+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:35.784 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:35.634+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:35.785 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:35.634+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:35.785 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:35.634+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:35.785 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:35.635+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:35.782+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:35.782+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:36.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:35.782+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:36.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:35.783+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:36.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:36.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:36.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:36.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:36.803+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:36.634+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:36.634+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:36.634+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:36.635+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:36.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:36.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:36.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:36.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:36.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:36.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:36.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:36.572+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:37.537+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:37.537+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:37.537+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:37.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:37.537+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:37.831 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:32:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:32:37.702+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29310 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:32:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:37.660+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:37.660+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:37.660+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:37.661+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:38.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:37.828+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:37.828+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:37.828+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:37.829+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:38.687+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:38.687+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:38.687+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:38.688+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:38.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:38.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:38.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:38.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:38.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:38.497+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:38.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:38.498+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:38.865+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:38.865+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:38.865+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:38.865+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:39.642+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:39.642+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:39.642+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:39.643+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:39.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:39.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:39.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:39.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:39.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:39.493+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:39.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:39.494+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:40.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:39.870+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:39.870+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:39.870+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:39.871+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:40.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:40.681+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:40.681+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:40.681+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:40.682+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:40.498+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:40.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:40.498+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:40.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:40.498+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:40.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:40.498+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:40.910+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:40.910+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:40.910+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:40.911+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:41.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:41.700+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:41.700+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:41.700+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:41.701+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:41.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:41.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:41.527+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:41.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:41.527+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:42.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:41.901+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:41.901+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:41.901+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:41.902+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:42.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:32:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:32:42.705+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29317 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:32:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:42.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:42.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:42.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:42.480+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:42.703+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:42.703+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:42.703+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:42.704+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:43.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:42.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:42.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:42.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:42.875+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:43.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:43.678+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:43.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:43.678+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:43.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:43.678+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:43.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:43.679+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:43.464+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:43.464+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:43.464+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:43.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:43.465+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:44.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:43.844+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:43.844+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:43.844+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:43.844+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:44.823 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:44.450+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:44.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:44.450+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:44.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:44.450+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:44.824 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:44.451+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:44.824 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:44.658+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:44.825 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:44.658+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:44.825 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:44.658+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:44.825 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:44.659+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:45.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:44.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:44.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:44.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:44.822+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:45.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:45.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:45.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:45.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:45.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:45.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:45.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:45.669+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:45.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:45.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:45.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:45.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:45.459+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:45.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:45.460+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:46.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:45.858+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:45.858+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:45.858+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:45.859+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:46.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:46.647+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:46.647+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:46.647+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:46.648+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:46.488+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:46.488+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:46.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:46.488+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:46.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:46.489+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:47.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:46.833+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:46.833+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:46.833+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:46.834+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:47.710 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:47.450+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:47.710 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:47.450+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:47.710 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:47.450+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:47.711 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:47.451+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:47.711 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:47.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:47.711 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:47.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:47.711 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:47.616+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:47.711 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:47.617+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:48.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:32:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:32:47.707+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29325 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:32:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:47.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:47.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:47.823+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:47.823+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:48.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:48.477+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:48.477+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:48.477+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:48.478+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:48.610+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:48.610+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:48.610+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:48.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:48.611+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:48.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:48.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:48.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:48.841+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:49.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:49.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:49.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:49.485+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:49.486+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:49.626+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:49.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:49.626+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:49.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:49.626+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:49.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:49.627+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:49.833+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:49.833+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:49.833+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:49.834+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:50.818 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:50.621+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:50.819 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:50.621+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:50.819 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:50.621+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:50.819 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:50.622+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:50.819 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:50.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:50.819 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:50.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:50.819 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:50.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:50.819 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:50.517+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:51.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:50.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:51.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:50.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:50.817+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:50.817+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:51.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:51.611+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:51.611+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:51.611+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:51.611+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:51.550+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:51.550+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:51.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:51.550+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:51.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:51.551+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:51.828+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:51.828+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:51.828+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:51.829+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:52.857 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:52.583+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:52.858 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:52.583+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:52.858 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:52.583+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:52.858 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:52.584+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:52.858 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:52.653+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:52.859 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:52.653+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:52.859 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:52.653+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:52.859 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:52.653+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:52.859 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:32:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:32:52.711+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29332 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:32:53.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:52.856+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:52.856+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:52.856+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:52.857+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:53.907 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:53.649+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:53.907 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:53.649+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:53.907 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:53.649+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:53.907 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:53.650+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:53.907 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:53.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:53.908 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:53.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:53.908 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:53.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:53.908 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:53.629+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:53.905+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:53.905+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:53.905+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:53.906+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:54.930 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:54.623+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:54.930 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:54.623+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:54.931 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:54.623+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:54.931 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:54.623+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:54.931 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:54.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:54.931 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:54.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:54.931 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:54.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:54.931 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:54.664+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:54.929+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:54.929+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:54.929+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:54.930+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:56.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:55.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:55.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:55.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:56.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:55.884+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:55.662+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:55.662+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:56.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:55.662+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:56.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:55.663+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:56.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:55.690+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:56.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:55.690+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:56.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:55.690+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:56.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:55.691+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:57.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:56.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:56.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:56.908+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:56.908+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:56.700+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:56.700+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:57.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:56.700+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:57.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:56.700+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:57.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:56.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:57.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:56.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:57.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:56.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:57.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:56.735+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:58.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:32:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:32:57.714+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29340 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:32:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:57.774+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:57.774+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:57.774+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:57.775+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:32:58.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:57.721+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:58.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:57.721+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:58.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:57.721+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:58.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:57.722+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:58.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:57.942+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:58.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:57.942+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:58.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:57.942+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:58.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:57.943+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:59.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:58.979+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:58.979+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:58.979+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:32:59.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:58.980+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:59.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:58.734+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:59.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:58.734+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:59.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:58.734+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:32:59.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:58.735+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:32:59.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:58.776+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:59.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:58.776+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:59.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:58.776+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:32:59.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:58.777+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:00.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:59.825+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:59.825+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:59.825+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:32:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:32:59.826+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:00.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:59.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:00.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:59.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:00.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:59.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:00.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:32:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:32:59.941+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:00.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:59.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:00.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:59.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:00.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:59.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:00.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:32:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:32:59.784+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:00.932+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:00.932+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:00.932+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:00.933+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:01.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:00.872+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:01.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:00.872+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:01.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:00.872+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:01.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:00.873+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:01.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:00.758+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:01.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:00.758+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:01.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:00.758+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:01.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:00.759+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:01.777+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:01.777+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:01.777+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:01.778+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:01.905+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:02.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:01.905+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:02.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:01.905+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:02.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:01.905+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:02.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:01.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:02.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:01.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:02.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:01.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:02.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:01.936+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:03.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:33:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:33:02.717+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29347 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:33:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:02.893+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:02.893+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:02.893+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:02.893+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:03.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:02.906+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:03.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:02.906+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:03.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:02.906+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:03.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:02.907+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:03.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:02.801+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:03.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:02.801+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:03.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:02.801+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:03.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:02.802+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:04.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:03.768+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:03.768+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:03.768+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:03.769+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:04.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:03.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:04.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:03.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:04.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:03.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:04.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:03.931+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:04.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:03.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:04.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:03.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:04.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:03.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:04.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:03.863+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:04.896 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:04.830+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:04.897 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:04.830+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:04.897 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:04.830+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:04.897 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:04.831+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:04.897 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:04.737+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:04.897 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:04.737+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:04.897 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:04.737+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:04.898 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:04.738+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:04.895+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:04.895+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:04.895+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:04.895+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:05.720+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:05.720+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:05.720+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:05.721+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:06.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:05.851+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:06.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:05.851+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:06.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:05.851+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:06.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:05.851+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:06.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:05.789+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:06.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:05.789+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:06.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:05.789+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:06.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:05.789+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:07.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:06.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:06.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:06.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:06.876+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:07.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:06.729+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:07.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:06.729+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:07.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:06.729+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:07.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:06.730+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:07.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:06.763+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:07.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:06.763+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:07.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:06.763+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:07.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:06.764+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:08.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:33:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:33:07.719+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29355 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:33:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:07.750+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:08.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:07.750+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:08.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:07.750+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:08.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:07.751+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:08.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:07.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:08.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:07.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:08.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:07.921+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:08.084 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:07.922+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:08.084 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:07.745+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:08.085 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:07.745+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:08.085 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:07.745+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:08.085 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:07.746+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:09.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:08.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:09.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:08.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:09.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:08.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:09.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:08.771+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:08.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:08.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:09.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:08.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:09.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:08.940+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:08.778+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:08.778+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:08.778+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:08.779+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:10.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:09.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:10.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:09.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:10.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:09.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:10.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:09.770+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:09.758+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:10.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:09.758+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:10.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:09.758+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:10.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:09.758+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:10.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:09.903+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:10.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:09.903+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:10.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:09.903+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:10.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:09.903+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:11.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:10.781+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:10.781+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:10.781+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:10.782+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:10.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:10.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:11.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:10.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:11.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:10.866+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:11.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:10.814+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:11.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:10.814+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:11.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:10.814+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:11.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:10.814+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:12.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:11.779+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:11.779+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:11.779+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:11.780+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:12.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:11.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:12.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:11.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:12.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:11.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:12.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:11.862+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:12.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:11.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:12.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:11.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:12.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:11.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:12.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:11.765+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:13.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:33:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:33:12.722+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29362 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:33:13.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:12.764+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:13.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:12.764+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:13.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:12.764+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:13.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:12.765+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:13.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:12.880+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:13.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:12.880+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:13.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:12.880+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:13.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:12.881+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:13.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:12.794+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:13.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:12.794+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:13.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:12.794+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:13.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:12.795+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:14.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:13.855+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:13.855+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:13.855+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:13.855+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:13.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:14.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:13.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:14.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:13.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:14.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:13.783+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:14.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:13.745+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:14.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:13.745+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:14.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:13.745+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:14.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:13.746+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:14.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:14.723+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:14.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:14.723+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:14.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:14.723+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:14.895 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:14.723+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:14.895 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:14.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:14.895 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:14.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:14.895 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:14.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:14.895 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:14.808+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:14.892+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:14.892+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:14.892+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:14.894+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:16.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:15.917+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:15.917+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:15.917+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:15.917+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:15.824+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:15.824+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:15.824+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:16.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:15.825+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:15.741+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:15.741+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:15.741+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:15.742+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:16.911 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:16.743+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:16.912 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:16.743+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:16.912 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:16.743+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:16.912 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:16.744+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:16.912 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:16.822+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:16.913 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:16.822+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:16.913 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:16.822+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:16.913 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:16.823+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:17.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:16.910+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:16.910+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:16.910+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:16.910+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:18.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:33:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:33:17.725+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29370 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:33:18.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:17.830+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:18.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:17.830+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:18.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:17.830+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:18.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:17.831+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:18.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:17.758+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:18.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:17.758+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:18.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:17.758+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:18.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:17.759+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:18.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:17.882+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:18.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:17.882+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:18.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:17.882+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:18.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:17.883+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:18.874+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:18.874+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:18.874+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:18.875+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:19.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:18.893+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:19.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:18.893+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:19.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:18.893+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:19.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:18.894+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:18.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:18.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:18.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:19.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:18.745+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:19.830+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:19.830+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:19.830+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:19.831+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:20.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:19.927+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:20.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:19.927+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:20.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:19.927+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:20.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:19.928+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:20.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:19.716+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:20.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:19.716+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:20.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:19.716+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:20.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:19.717+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:21.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:20.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:20.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:20.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:20.864+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:20.970+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:20.970+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:21.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:20.970+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:21.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:20.971+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:21.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:20.736+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:21.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:20.736+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:21.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:20.736+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:21.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:20.736+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:21.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:21.776+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:21.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:21.776+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:21.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:21.776+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:21.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:21.776+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:21.955 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:21.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:21.956 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:21.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:21.956 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:21.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:21.956 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:21.855+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:22.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:21.952+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:21.952+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:21.952+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:21.953+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:23.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:33:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:33:22.727+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29377 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:33:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:22.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:22.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:22.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:22.833+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:23.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:22.920+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:23.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:22.920+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:23.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:22.920+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:23.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:22.921+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:23.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:22.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:23.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:22.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:23.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:22.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:23.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:22.793+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:23.974 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:23.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:23.974 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:23.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:23.974 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:23.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:23.974 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:23.815+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:23.975 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:23.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:23.975 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:23.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:23.975 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:23.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:23.975 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:23.864+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:24.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:23.971+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:23.971+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:23.971+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:23.972+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:25.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:24.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:25.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:24.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:25.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:24.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:25.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:24.828+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:24.943+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:25.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:24.943+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:25.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:24.943+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:25.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:24.944+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:25.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:24.851+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:25.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:24.851+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:25.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:24.851+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:25.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:24.852+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:25.989+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:25.989+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:25.989+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:25.989+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:26.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:25.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:26.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:25.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:26.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:25.845+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:26.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:25.845+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:26.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:25.850+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:26.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:25.850+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:26.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:25.850+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:26.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:25.850+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:27.038 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:26.821+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:27.038 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:26.821+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:27.038 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:26.821+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:27.039 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:26.822+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:27.039 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:26.869+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:27.039 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:26.869+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:27.039 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:26.869+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:27.039 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:26.869+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:27.036+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:27.036+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:27.036+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:27.037+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:28.034 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:33:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:33:27.731+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29385 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:33:28.035 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:27.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:28.035 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:27.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:28.035 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:27.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:28.035 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:27.863+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:28.036 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:27.890+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:28.036 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:27.890+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:28.036 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:27.890+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:28.036 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:27.891+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:28.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:28.033+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:28.033+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:28.033+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:28.033+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:29.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:28.853+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:29.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:28.853+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:29.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:28.853+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:29.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:28.854+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:29.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:28.900+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:29.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:28.900+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:29.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:28.900+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:29.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:28.901+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:29.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:28.998+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:29.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:28.998+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:29.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:28.998+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:29.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:28.999+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:30.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:29.831+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:30.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:29.831+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:30.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:29.831+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:30.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:29.832+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:30.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:30.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:30.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:30.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:30.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:30.038+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:30.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:29.945+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:30.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:29.945+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:30.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:29.945+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:30.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:29.946+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:31.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:30.988+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:30.988+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:30.988+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:31.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:30.989+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:31.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:31.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:31.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:31.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:31.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:31.073+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:31.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:30.839+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:31.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:30.839+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:31.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:30.839+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:31.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:30.840+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:32.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:32.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:32.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:32.018+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:32.019+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:32.048+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:32.048+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:32.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:32.048+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:32.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:32.049+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:32.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:31.800+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:32.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:31.800+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:32.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:31.800+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:32.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:31.801+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:33.009 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:33:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:33:32.734+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29392 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:33:33.009 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:32.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:33.010 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:32.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:33.010 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:32.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:33.010 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:32.760+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:33.067+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:33.067+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:33.067+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:33.068+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:33.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:33.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:33.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:33.008+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:33.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:33.008+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:33.990 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:33.717+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:33.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:33.717+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:33.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:33.717+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:33.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:33.718+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:34.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:33.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:33.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:33.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:34.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:33.990+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:34.110+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:34.110+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:34.110+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:34.111+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:34.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:34.676+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:34.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:34.676+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:34.957 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:34.676+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:34.957 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:34.676+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:35.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:35.091+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:35.091+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:35.091+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:35.091+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:34.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:34.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:34.954+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:35.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:34.955+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:35.926 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:35.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:35.926 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:35.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:35.926 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:35.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:35.926 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:35.628+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:36.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:36.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:36.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:36.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:36.088+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:35.924+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:35.924+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:35.924+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:36.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:35.925+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:36.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:36.579+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:36.579+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:36.579+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:36.579+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:37.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:36.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:36.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:36.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:36.966+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:37.129+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:37.129+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:37.129+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:37.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:37.129+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:37.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:33:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:33:37.737+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29400 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:33:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:37.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:37.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:37.561+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:37.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:37.562+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:38.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:37.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:38.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:37.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:38.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:37.948+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:38.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:37.949+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:38.178+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:38.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:38.178+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:38.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:38.178+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:38.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:38.179+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:38.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:38.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:38.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:38.544+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:38.544+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:39.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:38.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:38.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:38.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:39.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:38.941+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:39.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:39.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:39.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:39.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:39.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:39.156+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:39.526+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:39.526+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:39.526+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:39.526+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:40.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:40.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:40.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:40.152+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:39.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:40.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:39.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:40.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:39.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:40.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:39.972+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:40.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:40.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:40.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:40.511+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:41.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:41.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:41.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:41.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:41.164+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:40.961+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:40.961+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:40.961+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:41.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:40.961+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:41.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:41.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:41.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:41.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:41.511+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:42.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:42.135+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:42.135+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:42.135+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:42.136+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:41.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:42.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:41.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:42.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:41.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:42.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:41.931+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:42.742 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:42.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:42.742 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:42.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:42.743 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:42.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:42.743 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:42.508+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:43.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:42.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:42.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:42.927+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:42.928+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:43.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:33:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:33:42.740+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29407 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:33:43.528 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:43.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:43.529 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:43.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:43.529 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:43.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:43.529 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:43.148+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:43.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:43.526+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:43.526+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:43.526+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:43.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:43.526+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:44.110+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:44.110+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:44.110+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:44.111+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:44.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:43.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:44.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:43.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:44.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:43.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:44.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:43.953+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:44.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:44.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:44.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:44.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:44.508+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:45.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:45.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:45.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:45.062+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:44.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:44.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:45.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:44.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:45.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:44.943+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:45.630 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:45.521+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:45.631 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:45.521+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:45.631 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:45.521+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:45.631 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:45.522+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:46.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:46.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:46.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:46.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:46.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:46.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:46.274 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:46.022+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:46.274 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:45.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:46.274 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:45.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:46.274 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:45.928+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:46.274 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:45.929+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:46.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:46.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:46.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:46.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:46.567+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:47.208 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:46.978+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:47.208 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:46.978+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:47.208 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:46.978+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:47.208 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:46.979+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:47.208 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:46.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:47.209 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:46.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:47.209 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:46.939+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:47.209 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:46.940+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:47.830 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:33:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:33:47.743+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29415 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:33:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:47.539+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:47.539+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:47.539+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:47.539+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:48.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:48.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:48.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:48.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:48.018+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:47.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:47.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:48.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:47.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:48.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:47.976+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:48.967 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:48.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:48.967 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:48.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:48.968 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:48.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:48.968 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:48.587+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:49.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:49.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:49.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:49.062+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:48.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:48.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:48.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:48.966+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:49.985 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:49.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:49.985 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:49.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:49.986 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:49.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:49.986 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:49.613+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:50.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:50.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:50.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:50.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:50.038+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:49.984+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:49.984+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:49.984+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:49.984+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:50.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:50.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:50.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:50.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:50.578+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:50.579+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:51.000+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:51.000+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:51.000+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:51.001+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:50.958+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:51.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:50.958+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:51.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:50.958+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:51.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:50.959+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:51.916 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:51.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:51.917 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:51.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:51.917 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:51.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:51.917 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:51.586+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:52.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:52.013+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:52.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:52.013+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:52.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:52.013+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:52.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:52.014+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:52.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:51.915+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:52.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:51.915+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:52.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:51.915+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:52.322 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:51.916+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:52.972 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:52.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:52.972 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:52.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:52.972 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:52.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:52.972 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:52.628+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:52.973 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:52.873+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:52.973 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:52.873+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:52.973 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:52.873+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:52.973 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:52.873+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:52.973 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:33:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:33:52.746+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29422 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:33:53.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:52.970+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:52.970+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:52.970+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:52.971+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:53.847 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:53.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:53.848 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:53.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:53.848 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:53.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:53.848 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:53.630+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:54.019+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:54.019+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:54.019+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:54.020+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:53.853+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:53.853+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:53.853+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:54.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:53.854+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:54.899 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:54.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:54.899 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:54.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:54.900 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:54.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:54.900 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:54.638+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:54.999+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:54.999+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:54.999+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:55.000+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:54.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:54.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:54.897+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:55.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:54.898+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:55.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:55.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:55.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:55.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:55.648+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:56.187 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:56.032+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:56.187 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:56.032+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:56.187 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:56.032+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:56.187 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:56.033+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:56.188 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:55.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:56.188 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:55.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:56.188 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:55.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:56.188 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:55.883+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:56.864 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:56.601+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:56.865 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:56.601+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:56.865 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:56.601+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:56.865 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:56.601+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:57.079+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:57.079+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:57.079+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:57.080+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:56.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:56.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:56.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:57.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:56.863+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:57.751 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:57.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:57.751 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:57.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:57.751 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:57.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:57.751 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:57.628+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:58.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:33:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:33:57.748+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29430 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:33:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:57.843+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:57.843+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:57.843+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:57.844+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:58.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:58.118+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:58.118+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:58.118+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:58.119+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:59.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:58.866+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:58.866+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:58.866+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:33:59.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:58.867+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:33:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:58.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:58.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:59.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:58.664+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:33:59.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:58.665+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:33:59.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:59.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:59.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:59.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:33:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:33:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:33:59.156+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:00.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:59.700+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:59.700+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:59.700+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:00.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:33:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:33:59.701+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:00.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:59.849+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:00.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:59.849+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:00.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:59.849+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:00.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:33:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:33:59.850+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:00.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:00.169+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:00.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:00.169+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:00.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:00.169+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:00.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:00.170+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:01.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:00.747+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:01.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:00.747+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:01.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:00.747+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:01.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:00.748+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:00.805+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:00.805+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:00.805+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:01.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:00.806+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:01.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:01.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:01.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:01.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:01.169+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:01.798+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:01.798+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:01.798+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:02.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:01.799+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:02.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:01.841+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:02.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:01.841+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:02.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:01.841+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:02.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:01.842+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:02.519 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:02.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:02.519 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:02.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:02.519 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:02.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:02.519 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:02.153+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:02.795 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:34:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:34:02.751+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29437 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:34:03.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:02.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:02.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:02.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:02.794+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:03.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:02.804+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:03.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:02.804+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:03.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:02.804+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:03.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:02.805+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:03.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:03.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:03.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:03.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:03.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:03.147+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:04.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:03.838+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:03.838+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:03.838+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:03.838+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:03.816+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:03.816+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:04.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:03.816+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:04.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:03.816+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:04.187+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:04.187+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:04.187+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:04.188+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:05.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:04.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:04.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:04.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:04.793+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:05.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:04.833+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:05.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:04.833+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:05.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:04.833+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:05.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:04.834+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:05.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:05.220+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:05.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:05.220+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:05.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:05.220+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:05.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:05.221+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:05.813+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:05.813+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:05.813+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:05.814+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:05.853+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:06.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:05.853+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:06.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:05.853+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:06.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:05.854+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:06.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:06.173+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:06.173+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:06.173+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:06.173+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:07.156 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:06.841+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:07.157 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:06.841+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:07.157 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:06.841+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:07.157 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:06.842+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:07.157 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:06.846+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:07.157 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:06.846+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:07.158 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:06.846+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:07.158 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:06.847+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:07.154+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:07.154+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:07.584 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:07.154+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:07.584 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:07.156+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:07.865+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:08.136 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:07.865+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:08.136 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:07.865+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:08.137 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:07.866+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:08.137 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:07.853+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:08.137 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:07.853+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:08.137 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:07.853+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:08.138 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:07.854+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:08.138 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:34:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:34:07.755+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29445 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:34:08.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:08.191+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:08.191+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:08.191+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:08.192+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:09.164 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:08.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:09.164 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:08.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:09.164 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:08.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:09.164 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:08.857+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:09.165 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:08.897+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:09.165 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:08.897+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:09.165 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:08.897+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:09.165 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:08.897+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:09.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:09.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:09.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:09.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:09.163+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:10.144 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:09.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:10.144 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:09.853+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:10.144 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:09.853+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:10.144 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:09.853+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:10.145 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:09.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:10.145 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:09.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:10.145 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:09.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:10.146 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:09.875+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:10.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:10.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:10.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:10.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:10.143+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:11.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:10.897+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:10.897+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:10.897+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:10.897+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:11.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:10.809+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:11.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:10.809+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:11.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:10.809+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:11.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:10.810+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:11.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:11.104+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:11.104+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:11.104+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:11.105+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:12.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:11.928+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:11.928+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:11.928+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:11.929+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:11.826+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:11.826+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:12.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:11.826+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:12.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:11.827+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:12.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:12.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:12.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:12.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:12.084+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:13.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:34:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:34:12.757+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29452 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:34:13.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:12.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:13.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:12.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:13.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:12.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:13.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:12.844+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:13.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:12.922+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:13.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:12.922+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:13.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:12.922+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:13.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:12.923+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:13.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:13.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:13.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:13.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:13.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:13.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:13.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:13.095+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:14.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:13.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:13.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:13.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:13.870+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:14.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:13.924+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:14.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:13.924+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:14.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:13.924+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:14.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:13.925+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:14.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:14.140+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:14.140+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:14.140+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:14.141+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:15.174 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:14.873+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:15.174 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:14.873+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:15.174 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:14.873+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:15.175 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:14.875+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:15.175 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:14.933+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:15.175 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:14.933+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:15.175 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:14.933+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:15.175 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:14.933+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:15.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:15.172+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:15.172+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:15.172+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:15.173+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:16.225 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:15.856+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:16.225 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:15.856+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:16.225 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:15.856+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:16.225 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:15.857+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:16.226 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:15.899+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:16.226 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:15.899+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:16.226 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:15.899+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:16.226 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:15.899+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:16.221+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:16.221+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:16.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:16.221+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:16.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:16.221+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:17.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:16.946+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:17.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:16.946+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:17.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:16.946+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:17.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:16.947+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:16.810+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:17.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:16.810+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:17.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:16.810+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:17.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:16.810+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:17.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:17.201+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:17.201+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:17.201+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:17.201+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:18.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:34:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:34:17.761+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29460 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:34:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:17.924+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:17.924+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:17.924+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:17.925+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:18.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:17.828+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:18.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:17.828+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:18.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:17.828+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:18.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:17.829+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:18.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:18.224+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:18.224+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:18.224+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:18.224+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:18.887+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:18.887+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:18.887+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:18.887+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:18.795+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:19.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:18.795+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:19.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:18.795+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:19.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:18.796+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:19.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:19.226+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:19.226+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:19.226+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:19.227+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:20.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:19.769+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:19.769+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:19.769+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:19.769+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:19.873+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:19.873+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:19.873+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:20.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:19.874+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:20.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:20.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:20.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:20.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:20.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:20.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:20.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:20.182+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:21.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:20.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:20.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:20.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:20.875+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:20.790+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:21.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:20.790+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:21.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:20.790+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:21.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:20.791+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:21.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:21.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:21.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:21.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:21.187+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:22.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:21.902+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:21.903+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:21.903+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:21.903+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:21.781+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:21.781+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:22.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:21.781+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:22.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:21.782+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:22.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:22.185+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:22.185+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:22.185+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:22.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:22.186+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:23.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:34:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:34:22.764+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29467 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:34:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:22.746+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:22.746+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:22.746+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:23.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:22.747+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:23.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:22.864+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:23.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:22.864+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:23.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:22.864+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:23.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:22.865+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:23.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:23.143+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:23.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:23.143+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:23.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:23.143+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:23.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:23.144+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:24.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:23.763+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:24.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:23.763+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:24.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:23.763+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:24.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:23.764+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:23.908+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:24.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:23.908+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:24.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:23.908+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:24.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:23.909+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:24.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:24.134+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:24.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:24.134+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:24.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:24.134+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:24.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:24.135+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:24.888 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:24.788+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:24.889 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:24.788+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:24.889 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:24.788+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:24.889 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:24.789+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:25.161 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:24.887+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:25.161 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:24.887+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:25.161 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:24.887+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:25.161 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:24.887+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:25.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:25.159+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:25.159+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:25.159+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:25.160+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:26.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:25.776+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:26.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:25.776+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:26.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:25.776+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:26.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:25.777+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:25.840+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:25.840+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:26.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:25.840+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:26.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:25.841+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:26.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:26.174+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:26.174+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:26.174+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:26.175+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:27.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:26.792+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:26.792+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:26.792+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:26.793+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:26.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:26.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:27.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:26.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:27.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:26.818+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:27.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:27.138+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:27.138+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:27.138+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:27.139+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:28.045 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:27.786+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:28.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:27.786+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:28.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:27.786+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:28.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:27.786+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:28.046 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:27.825+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:28.046 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:27.825+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:28.047 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:27.825+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:28.047 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:27.825+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:28.047 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:34:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:34:27.767+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29475 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:34:28.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:28.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:28.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:28.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:28.152+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:29.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:28.820+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:28.820+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:28.820+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:28.820+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:28.838+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:29.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:28.838+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:29.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:28.838+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:29.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:28.839+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:29.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:29.198+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:29.198+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:29.198+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:29.199+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:29.797+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:29.797+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:29.797+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:29.798+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:29.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:30.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:29.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:30.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:29.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:30.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:29.821+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:30.177+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:30.177+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:30.177+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:30.178+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:31.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:30.806+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:30.806+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:30.806+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:30.807+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:30.816+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:30.816+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:30.816+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:30.817+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:31.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:31.160+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:31.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:31.160+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:31.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:31.160+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:31.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:31.161+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:31.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:31.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:31.863+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:31.864+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:31.815+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:32.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:31.815+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:32.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:31.815+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:32.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:31.816+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:32.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:32.185+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:32.185+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:32.185+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:32.186+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:33.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:34:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:34:32.769+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29482 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:34:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:32.809+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:32.809+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:32.809+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:32.810+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:33.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:32.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:33.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:32.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:33.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:32.912+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:33.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:32.913+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:33.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:33.195+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:33.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:33.196+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:33.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:33.196+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:33.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:33.197+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:34.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:33.865+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:33.865+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:33.865+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:33.866+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:33.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:33.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:33.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:34.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:33.824+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:34.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:34.213+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:34.213+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:34.213+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:34.213+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:35.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:34.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:35.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:34.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:35.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:34.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:35.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:34.832+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:35.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:34.827+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:35.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:34.827+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:35.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:34.827+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:35.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:34.828+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:35.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:35.198+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:35.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:35.198+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:35.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:35.198+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:35.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:35.199+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:36.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:35.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:36.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:35.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:36.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:35.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:36.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:35.807+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:35.872+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:35.872+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:36.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:35.872+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:36.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:35.873+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:36.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:36.230+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:36.230+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:36.230+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:36.231+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:37.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:36.760+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:36.760+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:36.760+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:36.760+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:37.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:36.892+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:37.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:36.892+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:37.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:36.892+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:37.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:36.892+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:37.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:37.281+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:37.281+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:37.281+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:37.282+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:38.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:34:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:34:37.771+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29490 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:34:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:37.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:37.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:37.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:37.808+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:38.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:37.856+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:38.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:37.856+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:38.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:37.856+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:38.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:37.857+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:38.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:38.276+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:38.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:38.276+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:38.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:38.276+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:38.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:38.277+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:39.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:38.811+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:38.811+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:38.811+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:38.812+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:38.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:39.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:38.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:39.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:38.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:39.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:38.770+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:39.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:39.279+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:39.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:39.279+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:39.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:39.279+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:39.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:39.280+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:39.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:39.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:39.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:39.792+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:40.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:39.760+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:40.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:39.760+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:40.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:39.760+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:40.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:39.761+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:40.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:40.255+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:40.255+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:40.255+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:40.256+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:41.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:40.816+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:40.816+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:40.816+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:40.816+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:41.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:40.750+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:41.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:40.750+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:41.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:40.750+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:41.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:40.750+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:41.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:41.281+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:41.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:41.281+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:41.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:41.281+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:41.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:41.282+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:42.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:41.772+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:42.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:41.772+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:42.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:41.772+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:42.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:41.773+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:41.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:41.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:41.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:42.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:41.822+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:42.288+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:42.288+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:42.288+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:42.289+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:43.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:34:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:34:42.776+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29497 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:34:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:42.774+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:42.774+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:42.774+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:42.775+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:42.845+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:43.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:42.845+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:43.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:42.845+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:43.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:42.846+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:43.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:43.285+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:43.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:43.285+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:43.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:43.285+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:43.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:43.285+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:44.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:43.838+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:43.838+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:43.838+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:43.839+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:44.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:43.788+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:44.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:43.788+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:44.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:43.788+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:44.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:43.789+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:44.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:44.251+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:44.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:44.251+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:44.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:44.251+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:44.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:44.252+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:44.884 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:44.741+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:44.885 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:44.741+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:44.885 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:44.741+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:44.885 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:44.742+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:45.224 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:44.883+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:45.224 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:44.883+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:45.225 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:44.883+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:45.225 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:44.884+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:45.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:45.223+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:45.223+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:45.223+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:45.223+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:45.841 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:45.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:45.841 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:45.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:45.841 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:45.765+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:45.842 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:45.766+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:46.239 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:45.839+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:46.240 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:45.839+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:46.240 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:45.839+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:46.240 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:45.840+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:46.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:46.238+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:46.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:46.238+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:46.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:46.238+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:46.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:46.238+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:47.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:46.850+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:47.085 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:46.850+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:47.085 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:46.850+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:47.085 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:46.851+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:47.085 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:46.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:47.086 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:46.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:47.086 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:46.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:47.086 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:46.767+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:47.198+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:47.198+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:47.198+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:47.199+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:48.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:34:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:34:47.779+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29505 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:34:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:47.768+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:47.768+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:47.768+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:47.769+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:48.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:47.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:48.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:47.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:48.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:47.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:48.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:47.875+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:48.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:48.220+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:48.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:48.220+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:48.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:48.220+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:48.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:48.220+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:49.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:48.759+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:48.759+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:48.759+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:48.760+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:48.915+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:48.915+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:49.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:48.915+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:49.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:48.916+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:49.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:49.179+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:49.179+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:49.179+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:49.179+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:50.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:49.796+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:50.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:49.796+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:50.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:49.796+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:50.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:49.797+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:49.908+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:49.908+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:50.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:49.908+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:50.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:49.909+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:50.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:50.193+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:50.193+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:50.193+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:50.194+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:51.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:50.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:51.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:50.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:51.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:50.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:51.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:50.784+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:51.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:50.916+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:51.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:50.916+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:51.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:50.916+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:51.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:50.917+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:51.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:51.212+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:51.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:51.212+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:51.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:51.212+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:51.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:51.213+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:52.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:51.963+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:52.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:51.963+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:52.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:51.963+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:52.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:51.964+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:51.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:51.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:52.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:51.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:52.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:51.828+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:52.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:52.241+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:52.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:52.241+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:52.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:52.241+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:52.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:52.242+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:53.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:52.988+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:53.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:52.988+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:53.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:52.988+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:53.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:52.988+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:52.789+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:53.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:52.789+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:53.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:52.789+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:53.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:52.790+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:53.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:34:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:34:52.781+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29512 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:34:53.567 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:53.211+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:53.567 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:53.211+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:53.567 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:53.211+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:53.567 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:53.212+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:53.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:53.744+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:53.744+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:53.744+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:53.744+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:54.170 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:54.026+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:54.170 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:54.026+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:54.170 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:54.026+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:54.170 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:54.027+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:54.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:54.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:54.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:54.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:54.169+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:54.996 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:54.744+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:54.996 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:54.744+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:54.996 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:54.744+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:54.996 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:54.744+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:55.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:54.994+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:55.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:54.994+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:55.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:54.994+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:55.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:54.995+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:55.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:55.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:55.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:55.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:55.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:55.167+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:56.037 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:55.781+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:56.037 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:55.781+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:56.037 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:55.781+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:56.037 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:55.782+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:56.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:56.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:56.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:56.150+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:56.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:56.035+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:56.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:56.035+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:56.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:56.035+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:56.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:56.036+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:57.037 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:56.776+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:57.037 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:56.776+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:57.037 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:56.776+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:57.037 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:56.777+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:57.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:57.035+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:57.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:57.035+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:57.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:57.035+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:57.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:57.036+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:57.148+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:57.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:57.148+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:57.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:57.148+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:57.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:57.149+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:58.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:58.026+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:58.026+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:58.026+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:58.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:58.027+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:34:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:57.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:57.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:58.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:57.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:58.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:57.784+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:58.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:34:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:34:57.784+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29520 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:34:58.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:58.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:58.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:58.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:58.195+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:59.038 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:58.756+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:59.038 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:58.756+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:59.038 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:58.756+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:34:59.038 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:58.758+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:59.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:59.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:59.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:59.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:34:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:34:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:34:59.150+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:34:59.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:59.036+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:59.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:59.036+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:59.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:59.036+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:34:59.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:34:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:34:59.037+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:00.050 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:59.720+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:00.050 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:59.720+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:00.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:59.720+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:00.051 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:34:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:34:59.721+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:00.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:00.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:00.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:00.125+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:00.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:00.048+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:00.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:00.048+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:00.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:00.048+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:00.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:00.049+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:01.028 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:00.743+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:01.029 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:00.743+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:01.029 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:00.743+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:01.029 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:00.743+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:01.084+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:01.084+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:01.084+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:01.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:01.084+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:01.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:01.027+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:01.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:01.027+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:01.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:01.027+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:01.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:01.028+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:02.064 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:01.786+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:02.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:01.786+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:02.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:01.786+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:02.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:01.787+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:02.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:02.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:02.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:02.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:02.063+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:02.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:02.074+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:02.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:02.074+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:02.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:02.074+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:02.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:02.075+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:03.026 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:35:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:35:02.787+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29527 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:35:03.027 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:02.773+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:03.027 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:02.773+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:03.027 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:02.773+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:03.027 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:02.774+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:03.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:03.047+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:03.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:03.047+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:03.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:03.047+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:03.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:03.048+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:03.025+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:03.025+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:03.025+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:03.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:03.025+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:04.044 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:03.724+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:04.045 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:03.724+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:04.045 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:03.724+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:04.045 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:03.725+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:04.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:04.046+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:04.046+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:04.046+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:04.047+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:04.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:04.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:04.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:04.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:04.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:04.044+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:05.002 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:04.709+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:05.003 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:04.709+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:05.003 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:04.709+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:05.003 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:04.709+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:05.009+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:05.009+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:05.009+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:05.010+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:05.001+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:05.001+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:05.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:05.001+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:05.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:05.002+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:05.958 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:05.659+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:05.958 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:05.659+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:05.958 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:05.659+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:05.958 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:05.660+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:06.050+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:06.050+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:06.050+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:06.051+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:06.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:05.956+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:06.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:05.956+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:06.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:05.956+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:06.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:05.957+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:07.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:06.689+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:07.079 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:06.689+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:07.079 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:06.689+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:07.079 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:06.690+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:07.079 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:06.912+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:07.079 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:06.912+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:07.079 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:06.912+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:07.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:06.913+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:07.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:07.076+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:07.077+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:07.077+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:07.077+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:08.037 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:35:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:35:07.790+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29535 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:35:08.037 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:07.714+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:08.037 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:07.714+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:08.037 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:07.714+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:08.038 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:07.715+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:08.038 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:07.958+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:08.039 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:07.958+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:08.039 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:07.958+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:08.039 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:07.959+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:08.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:08.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:08.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:08.036+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:08.948 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:08.667+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:08.948 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:08.667+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:08.949 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:08.667+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:08.949 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:08.668+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:08.947+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:08.947+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:08.947+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:08.947+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:08.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:09.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:08.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:09.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:08.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:09.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:08.990+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:09.996 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:09.698+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:09.997 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:09.698+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:09.997 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:09.698+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:09.997 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:09.699+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:09.997 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:09.906+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:09.997 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:09.906+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:09.998 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:09.906+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:09.998 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:09.906+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:10.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:09.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:09.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:09.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:09.995+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:11.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:10.956+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:10.956+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:10.956+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:10.957+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:11.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:10.738+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:11.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:10.738+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:11.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:10.738+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:11.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:10.738+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:11.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:10.933+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:11.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:10.933+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:11.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:10.933+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:11.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:10.933+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:12.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:11.992+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:12.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:11.992+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:12.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:11.992+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:12.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:11.993+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:11.769+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:12.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:11.769+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:12.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:11.769+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:12.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:11.770+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:12.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:11.917+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:12.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:11.917+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:12.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:11.917+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:12.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:11.917+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:13.031 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:12.885+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:13.031 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:12.885+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:13.031 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:12.885+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:13.031 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:12.886+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:13.032 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:12.756+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:13.032 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:12.756+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:13.032 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:12.756+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:13.032 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:12.756+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:13.032 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:35:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:35:12.792+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29542 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:35:13.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:13.028+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:13.028+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:13.028+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:13.029+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:14.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:13.864+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:14.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:13.864+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:14.001 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:13.864+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:14.002 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:13.865+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:14.002 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:13.753+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:14.002 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:13.753+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:14.002 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:13.753+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:14.002 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:13.754+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:14.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:13.998+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:13.998+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:13.998+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:13.999+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:14.864 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:14.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:14.865 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:14.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:14.865 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:14.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:14.865 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:14.754+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:14.956+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:14.956+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:14.956+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:47.433595+0000 (oldest deadline 2024-09-10T04:25:07.433595+0000) 2024-09-10T04:35:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:14.957+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:14.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:15.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:14.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:15.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:14.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:15.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:14.863+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:16.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:15.974+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:15.768+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:15.768+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:15.768+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:15.769+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:16.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:15.830+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:15.830+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:15.830+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:15.830+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:17.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:16.960+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:16.752+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:16.752+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:16.752+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:16.753+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:17.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:16.836+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:17.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:16.836+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:17.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:16.836+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:17.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:16.836+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:17.832+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:17.832+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:17.832+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:18.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:17.833+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:18.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:17.942+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:18.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:17.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:18.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:17.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:18.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:17.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:18.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:17.755+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:18.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:35:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:35:17.795+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29550 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:35:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:18.905+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:18.830+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:18.830+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:18.830+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:18.831+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:19.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:18.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:19.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:18.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:19.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:18.770+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:19.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:18.771+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:19.938+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:19.878+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:19.878+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:19.878+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:19.879+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:20.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:19.774+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:20.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:19.774+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:20.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:19.774+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:20.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:19.774+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:21.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:20.932+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:20.870+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:20.870+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:20.870+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:20.871+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:21.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:20.733+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:21.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:20.733+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:21.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:20.733+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:21.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:20.733+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:22.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:21.949+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:21.865+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:21.865+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:21.865+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:22.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:21.866+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:22.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:21.768+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:22.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:21.768+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:22.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:21.768+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:22.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:21.769+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:23.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:22.975+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:22.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:22.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:22.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:23.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:22.824+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:23.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:22.738+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:23.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:22.738+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:23.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:22.738+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:23.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:22.738+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:23.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:35:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:35:22.799+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29557 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:35:24.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:23.958+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:23.800+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:23.800+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:23.800+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:23.801+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:24.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:23.725+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:24.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:23.725+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:24.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:23.725+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:24.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:23.725+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:24.755+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:24.755+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:24.755+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:24.755+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:24.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:24.677+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:24.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:24.677+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:24.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:24.677+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:24.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:24.678+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:25.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:24.930+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:26.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:25.928+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:25.753+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:25.753+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:25.753+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:51.701384+0000 (oldest deadline 2024-09-10T04:25:11.701384+0000) 2024-09-10T04:35:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:25.754+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:26.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:25.704+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:26.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:25.704+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:26.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:25.704+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:26.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:25.705+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:27.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:26.895+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:26.738+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:26.708+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:26.708+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:26.708+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:27.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:26.708+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:28.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:35:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:35:27.802+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29565 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:35:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:27.889+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:28.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:27.768+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:28.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:27.680+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:28.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:27.680+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:28.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:27.680+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:28.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:27.680+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:29.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:28.919+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:28.720+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:28.694+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:28.694+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:29.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:28.694+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:29.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:28.695+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:29.935+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:29.768+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:29.699+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:29.699+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:30.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:29.699+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:30.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:29.699+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:30.914+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:30.795+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:30.684+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:30.684+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:30.684+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:31.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:30.685+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:32.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:31.891+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:31.822+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:31.706+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:31.706+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:31.706+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:32.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:31.706+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:33.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:32.889+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:32.861+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:33.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:32.688+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:33.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:32.688+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:33.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:32.688+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:33.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:32.689+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:33.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:35:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:35:32.805+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29572 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:35:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:33.914+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:33.838+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:34.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:33.730+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:34.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:33.730+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:34.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:33.730+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:24:56.207957+0000 (oldest deadline 2024-09-10T04:25:16.207957+0000) 2024-09-10T04:35:34.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:33.730+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:35.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:34.929+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:35.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:34.771+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:34.807+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:36.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:35.935+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:36.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:35.730+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:35.801+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:37.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:36.726+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:37.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:36.847+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:36.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:36.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:36.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:37.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:36.931+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:38.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:35:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:35:37.807+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29580 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:35:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:37.775+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:37.865+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:37.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:38.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:37.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:38.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:37.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:38.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:37.941+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:39.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:38.792+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:38.834+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:38.951+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:38.951+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:39.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:38.951+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:39.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:38.952+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:39.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:39.757+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:40.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:39.848+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:39.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:39.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:39.904+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:39.904+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:41.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:40.833+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:41.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:40.716+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:40.913+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:40.913+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:40.913+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:41.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:40.914+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:42.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:41.707+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:41.872+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:42.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:41.877+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:42.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:41.877+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:42.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:41.877+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:42.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:41.877+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:43.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:35:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:35:42.810+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29587 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:35:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:42.716+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:42.840+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:42.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:43.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:42.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:43.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:42.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:43.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:42.836+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:44.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:43.695+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:43.838+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:43.825+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:43.825+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:43.825+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:44.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:43.826+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:44.873 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:44.813+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:44.873 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:44.726+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:44.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:44.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:44.871+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:44.872+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:46.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:45.788+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:45.847+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:45.847+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:45.847+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:45.848+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:46.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:45.702+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:47.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:46.756+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:46.858+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:46.858+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:46.858+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:46.859+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:47.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:46.722+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:48.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:35:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:35:47.813+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29591 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:35:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:47.705+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:47.802+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:47.880+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:48.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:47.880+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:48.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:47.880+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:48.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:47.881+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:49.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:48.670+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:48.852+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:49.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:48.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:49.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:48.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:49.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:48.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:49.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:48.876+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:50.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:49.655+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:49.886+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:49.886+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:49.886+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:50.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:49.886+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:50.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:49.820+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:50.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:49.820+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:50.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:49.820+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:50.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:49.821+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:51.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:50.633+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:50.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:50.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:50.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:50.841+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:51.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:50.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:51.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:50.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:51.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:50.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:51.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:50.863+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:52.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:51.654+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:51.846+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:51.846+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:51.846+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:51.846+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:52.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:51.872+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:52.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:51.872+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:52.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:51.872+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:52.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:51.873+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:52.870 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:52.857+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:52.870 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:52.857+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:52.870 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:52.857+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:52.870 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:52.858+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:52.871 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:52.617+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:52.871 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:35:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:35:52.815+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29598 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:35:53.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:52.867+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:53.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:52.867+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:53.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:52.867+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:53.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:52.868+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:53.869 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:53.581+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:53.870 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:53.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:53.870 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:53.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:53.870 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:53.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:53.870 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:53.821+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:53.867+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:53.867+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:53.867+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:53.868+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:54.870 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:54.628+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:54.871 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:54.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:54.871 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:54.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:54.871 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:54.817+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:54.871 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:54.818+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:55.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:54.867+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:55.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:54.867+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:55.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:54.867+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:55.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:54.868+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:55.877 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:55.602+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:55.877 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:55.785+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:55.878 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:55.785+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:55.878 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:55.785+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:55.878 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:55.786+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:55.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:55.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:55.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:55.876+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:56.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:56.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:56.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:56.738+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:56.571+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:56.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:56.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:56.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:57.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:56.902+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:57.822 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:57.691+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:57.822 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:57.691+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:57.822 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:57.691+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:57.822 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:57.691+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:57.823 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:35:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:35:57.820+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29606 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:35:57.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:57.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:35:57.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:57.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:35:57.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:57.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:35:57.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:57.529+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:58.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:57.939+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:57.939+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:57.939+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:57.940+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:58.726+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:58.726+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:58.726+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:58.726+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:35:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:58.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:35:58.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:58.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:35:58.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:58.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:35:58.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:58.534+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:59.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:58.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:58.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:58.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:35:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:35:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:35:58.989+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:35:59.746 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:59.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:35:59.746 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:59.494+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:35:59.746 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:59.494+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:35:59.747 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:59.494+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:35:59.747 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:35:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:35:59.495+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:00.025 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:59.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:00.025 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:59.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:00.025 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:35:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:35:59.745+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:00.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:00.023+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:00.023+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:00.023+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:00.024+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:00.751 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:00.491+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:00.752 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:00.491+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:00.752 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:00.491+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:00.752 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:00.492+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:00.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:00.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:00.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:00.975+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:01.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:00.750+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:01.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:00.750+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:01.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:00.750+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:01.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:00.750+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:01.798 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:01.482+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:01.799 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:01.482+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:01.799 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:01.482+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:01.799 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:01.482+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:02.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:01.986+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:01.986+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:01.986+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:02.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:01.987+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:01.797+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:01.797+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:01.797+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:02.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:01.797+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:02.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:02.466+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:02.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:02.466+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:02.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:02.466+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:02.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:02.467+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:03.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:02.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:02.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:02.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:03.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:02.948+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:03.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:02.752+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:03.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:02.752+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:03.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:02.752+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:03.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:02.753+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:03.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:36:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:36:02.823+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29613 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:36:03.730 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:03.446+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:03.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:03.446+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:03.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:03.446+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:03.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:03.447+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:03.992 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:03.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:03.993 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:03.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:03.993 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:03.731+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:03.993 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:03.732+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:04.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:03.991+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:03.991+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:03.991+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:04.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:03.992+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:04.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:04.452+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:04.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:04.452+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:04.778 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:04.452+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:04.778 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:04.453+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:04.775+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:04.775+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:04.775+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:04.776+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:04.948+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:05.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:04.948+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:05.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:04.948+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:05.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:04.949+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:05.736 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:05.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:05.736 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:05.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:05.736 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:05.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:05.736 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:05.416+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:05.989 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:05.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:05.990 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:05.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:05.990 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:05.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:05.990 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:05.735+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:06.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:05.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:05.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:05.988+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:05.989+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:06.746 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:06.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:06.746 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:06.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:06.746 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:06.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:06.746 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:06.412+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:06.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:06.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:06.744+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:06.745+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:06.959+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:07.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:06.959+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:07.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:06.959+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:07.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:06.959+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:07.711 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:07.447+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:07.712 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:07.447+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:07.712 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:07.447+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:07.712 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:07.447+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:08.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:07.709+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:07.709+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:07.709+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:07.710+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:07.919+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:08.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:07.919+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:08.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:07.919+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:08.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:07.920+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:08.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:36:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:36:07.826+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29621 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:36:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:08.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:08.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:08.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:08.669+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:08.471+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:08.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:08.472+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:08.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:08.472+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:08.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:08.472+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:09.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:08.969+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:08.969+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:08.969+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:09.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:08.970+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:09.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:09.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:09.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:09.703+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:09.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:09.497+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:09.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:09.497+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:09.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:09.497+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:09.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:09.498+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:10.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:09.996+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:09.996+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:09.996+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:09.996+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:10.725+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:10.725+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:10.725+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:10.726+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:10.523+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:10.523+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:10.523+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:10.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:10.523+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:10.977+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:10.977+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:10.977+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:10.978+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:11.680+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:11.680+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:11.680+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:11.681+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:11.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:11.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:11.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:11.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:11.563+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:12.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:12.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:12.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:12.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:12.005+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:13.044 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:36:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:36:12.829+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29628 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:36:13.045 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:12.598+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:13.045 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:12.598+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:13.045 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:12.598+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:13.045 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:12.599+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:13.045 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:12.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:13.045 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:12.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:13.045 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:12.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:13.046 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:12.659+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:13.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:13.042+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:13.042+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:13.042+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:13.043+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:14.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:13.618+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:14.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:13.618+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:14.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:13.618+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:14.017 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:13.619+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:14.018 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:13.613+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:14.018 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:13.613+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:14.018 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:13.613+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:14.018 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:13.614+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:14.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:14.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:14.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:14.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:14.016+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:14.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:14.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:14.618+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:14.619+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:14.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:14.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:14.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:14.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:14.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:14.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:14.578+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:15.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:15.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:15.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:15.017+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:16.009 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:15.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:16.010 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:15.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:16.010 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:15.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:16.010 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:15.610+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:16.010 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:15.666+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:16.010 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:15.666+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:16.011 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:15.666+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:16.011 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:15.666+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:16.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:16.008+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:16.008+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:16.008+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:16.008+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:16.992 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:16.631+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:16.992 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:16.631+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:16.993 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:16.631+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:16.993 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:16.631+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:16.993 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:16.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:16.993 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:16.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:16.993 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:16.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:16.994 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:16.628+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:16.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:16.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:16.990+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:16.991+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:17.949 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:17.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:17.949 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:17.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:17.949 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:17.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:17.950 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:17.647+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:17.950 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:36:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:36:17.832+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29636 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:36:17.951 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:17.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:17.951 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:17.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:17.951 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:17.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:17.951 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:17.668+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:18.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:17.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:17.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:17.947+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:17.948+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:18.902 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:18.637+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:18.902 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:18.637+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:18.902 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:18.637+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:18.902 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:18.638+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:18.903 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:18.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:18.903 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:18.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:18.903 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:18.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:18.903 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:18.614+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:19.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:18.900+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:18.900+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:18.900+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:18.901+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:19.924 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:19.622+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:19.924 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:19.622+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:19.924 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:19.622+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:19.924 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:19.623+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:19.925 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:19.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:19.925 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:19.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:19.925 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:19.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:19.925 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:19.611+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:20.285 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:19.922+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:20.285 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:19.922+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:20.285 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:19.922+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:20.285 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:19.923+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:20.878 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:20.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:20.879 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:20.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:20.879 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:20.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:20.879 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:20.583+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:20.879 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:20.641+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:20.879 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:20.641+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:20.880 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:20.641+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:20.880 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:20.642+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:21.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:20.877+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:21.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:20.877+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:21.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:20.877+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:21.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:20.877+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:21.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:21.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:21.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:21.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:21.556+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:21.640+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:21.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:21.640+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:21.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:21.640+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:21.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:21.640+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:22.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:21.838+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:21.838+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:21.838+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:21.838+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:22.855 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:22.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:22.855 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:22.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:22.855 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:22.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:22.855 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:22.604+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:22.856 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:22.596+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:22.856 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:22.596+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:22.856 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:22.596+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:22.856 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:22.597+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:22.856 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:36:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:36:22.835+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29643 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:36:23.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:22.853+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:22.853+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:22.853+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:22.854+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:23.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:23.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:23.568+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:23.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:23.568+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:23.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:23.609+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:23.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:23.609+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:23.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:23.609+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:23.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:23.609+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:23.869+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:23.869+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:23.869+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:23.870+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:24.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:24.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:24.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:24.624+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:24.625+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:24.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:24.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:24.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:24.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:24.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:24.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:24.554+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:25.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:24.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:24.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:24.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:24.837+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:25.657 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:25.572+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:25.657 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:25.572+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:25.657 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:25.572+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:25.658 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:25.573+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:26.016 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:25.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:26.017 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:25.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:26.017 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:25.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:26.017 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:25.821+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:26.017 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:25.655+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:26.017 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:25.655+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:26.018 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:25.655+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:26.018 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:25.656+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:26.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:26.695+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:26.695+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:26.695+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:26.696+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:26.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:26.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:26.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:26.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:26.535+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:26.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:26.536+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:27.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:26.861+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:26.861+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:26.861+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:26.862+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:27.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:27.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:27.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:27.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:27.546+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:27.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:27.652+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:27.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:27.652+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:27.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:27.652+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:27.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:27.653+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:28.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:36:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:36:27.837+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29651 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:36:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:27.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:27.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:27.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:27.903+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:28.897 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:28.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:28.897 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:28.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:28.897 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:28.595+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:28.897 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:28.596+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:28.898 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:28.679+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:28.898 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:28.679+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:28.898 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:28.679+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:28.898 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:28.680+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:29.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:28.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:28.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:28.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:28.895+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:29.932 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:29.689+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:29.933 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:29.689+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:29.933 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:29.689+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:29.933 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:29.689+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:29.933 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:29.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:29.933 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:29.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:29.933 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:29.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:29.934 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:29.627+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:29.929+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:29.929+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:29.929+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:30.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:29.930+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:30.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:30.682+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:30.682+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:30.682+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:30.683+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:30.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:30.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:30.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:30.652+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:30.917+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:30.917+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:30.917+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:30.918+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:31.902 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:31.714+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:31.903 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:31.714+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:31.903 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:31.714+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:31.903 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:31.715+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:31.903 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:31.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:31.904 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:31.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:31.904 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:31.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:31.904 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:31.629+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:31.904 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:31.899+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:31.904 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:31.899+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:31.905 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:31.899+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:32.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:31.900+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:32.941 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:36:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:36:32.840+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29658 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:36:32.942 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:32.721+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:32.942 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:32.721+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:32.942 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:32.721+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:32.942 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:32.722+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:32.942 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:32.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:32.943 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:32.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:32.943 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:32.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:32.943 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:32.659+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:32.938+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:32.938+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:32.938+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:32.939+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:33.962 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:33.772+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:33.962 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:33.772+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:33.963 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:33.772+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:33.963 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:33.772+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:33.963 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:33.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:33.963 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:33.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:33.963 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:33.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:33.963 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:33.629+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:34.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:33.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:33.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:33.960+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:33.961+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:34.979 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:34.793+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:34.979 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:34.793+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:34.979 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:34.793+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:34.979 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:34.793+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:34.980 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:34.606+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:34.980 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:34.606+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:34.980 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:34.606+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:34.980 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:34.607+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:35.232 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:34.977+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:35.233 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:34.977+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:35.233 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:34.977+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:35.233 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:34.977+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:35.829 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:35.639+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:35.829 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:35.639+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:35.639+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:35.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:35.639+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:36.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:35.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:36.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:35.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:36.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:35.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:36.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:35.828+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:35.986+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:36.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:35.986+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:36.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:35.986+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:36.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:35.986+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:36.858 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:36.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:36.859 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:36.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:36.859 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:36.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:36.859 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:36.603+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:37.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:36.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:36.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:36.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:36.958+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:36.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:37.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:36.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:37.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:36.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:37.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:36.857+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:37.945 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:37.812+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:37.945 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:37.812+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:37.945 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:37.812+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:37.945 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:37.813+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:37.946 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:37.599+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:37.946 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:37.599+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:37.946 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:37.599+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:37.946 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:37.600+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:37.947 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:36:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:36:37.844+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29666 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:36:38.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:37.943+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:38.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:37.943+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:37.943+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:37.944+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:38.938 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:38.842+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:38.938 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:38.842+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:38.938 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:38.842+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:38.938 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:38.843+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:38.939 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:38.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:38.939 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:38.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:38.939 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:38.598+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:38.939 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:38.599+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:38.936+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:38.936+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:38.936+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:38.937+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:39.897 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:39.851+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:39.897 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:39.851+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:39.897 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:39.851+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:39.897 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:39.852+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:39.897 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:39.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:39.898 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:39.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:39.898 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:39.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:39.898 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:39.633+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:40.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:39.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:39.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:39.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:39.895+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:40.857+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:40.857+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:40.857+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:40.858+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:41.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:40.812+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:41.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:40.812+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:41.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:40.812+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:41.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:40.813+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:41.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:40.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:41.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:40.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:41.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:40.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:41.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:40.680+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:42.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:41.809+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:42.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:41.809+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:42.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:41.809+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:42.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:41.810+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:42.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:41.846+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:42.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:41.846+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:42.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:41.846+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:42.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:41.847+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:42.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:41.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:42.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:41.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:42.084 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:41.666+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:42.084 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:41.667+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:43.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:36:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:36:42.847+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29673 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:36:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:42.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:42.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:42.679+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:43.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:42.680+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:43.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:42.815+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:43.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:42.815+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:43.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:42.815+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:43.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:42.816+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:43.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:42.868+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:43.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:42.868+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:43.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:42.868+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:43.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:42.869+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:43.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:43.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:43.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:43.696+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:44.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:43.851+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:44.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:43.851+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:44.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:43.851+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:44.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:43.852+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:44.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:43.802+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:44.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:43.802+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:44.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:43.802+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:44.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:43.802+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:44.827 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:44.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:44.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:44.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:44.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:44.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:44.828 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:44.662+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:44.825+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:44.825+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:44.825+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:45.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:44.826+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:45.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:44.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:45.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:44.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:45.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:44.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:45.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:44.853+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:46.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:45.800+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:45.800+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:45.800+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:46.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:45.801+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:46.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:45.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:46.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:45.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:46.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:45.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:46.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:45.895+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:46.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:45.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:46.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:45.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:46.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:45.698+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:46.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:45.699+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:46.931 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:46.673+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:46.931 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:46.673+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:46.931 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:46.673+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:46.931 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:46.673+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:46.932 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:46.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:46.932 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:46.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:46.932 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:46.799+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:46.932 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:46.800+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:47.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:46.929+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:46.929+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:46.929+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:46.930+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:47.939 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:47.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:47.939 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:47.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:47.939 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:47.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:47.940 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:47.821+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:47.940 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:47.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:47.940 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:47.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:47.940 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:47.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:47.940 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:47.630+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:47.941 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:36:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:36:47.850+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29681 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:36:48.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:47.937+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:47.937+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:47.937+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:48.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:47.938+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:48.916 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:48.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:48.917 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:48.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:48.917 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:48.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:48.917 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:48.606+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:48.918 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:48.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:48.918 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:48.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:48.918 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:48.816+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:48.919 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:48.817+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:49.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:48.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:48.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:48.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:48.916+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:49.917 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:49.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:49.918 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:49.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:49.918 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:49.591+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:49.918 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:49.592+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:49.918 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:49.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:49.918 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:49.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:49.919 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:49.775+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:49.919 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:49.775+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:50.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:49.916+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:49.916+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:49.916+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:49.917+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:50.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:50.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:50.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:50.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:50.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:50.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:50.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:50.628+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:50.895 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:50.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:50.895 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:50.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:50.895 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:50.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:50.895 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:50.759+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:51.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:50.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:50.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:50.914+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:50.915+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:51.965 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:51.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:51.965 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:51.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:51.965 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:51.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:51.966 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:51.589+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:51.966 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:51.763+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:51.966 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:51.763+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:51.966 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:51.763+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:51.966 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:51.764+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:52.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:51.963+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:51.963+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:51.963+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:51.964+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:52.810+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:52.810+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:52.810+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:52.811+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:52.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:52.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:52.566+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:52.567+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:53.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:36:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:36:52.852+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29688 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:36:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:52.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:52.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:52.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:53.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:52.953+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:53.920 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:53.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:53.920 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:53.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:53.921 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:53.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:53.921 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:53.583+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:53.921 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:53.826+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:53.921 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:53.826+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:53.921 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:53.826+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:53.921 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:53.827+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:54.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:53.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:53.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:53.918+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:54.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:53.919+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:54.937 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:54.787+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:54.937 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:54.787+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:54.937 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:54.787+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:54.937 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:54.788+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:54.937 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:54.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:54.938 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:54.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:54.938 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:54.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:54.938 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:54.633+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:55.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:54.935+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:54.935+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:54.935+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:54.936+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:55.981 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:55.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:55.981 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:55.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:55.981 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:55.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:55.981 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:55.810+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:55.982 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:55.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:55.982 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:55.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:55.982 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:55.920+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:55.982 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:55.921+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:55.983 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:55.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:55.983 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:55.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:55.983 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:55.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:55.983 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:55.672+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:56.955 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:56.764+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:56.955 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:56.764+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:56.955 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:56.764+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:56.955 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:56.764+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:56.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:56.652+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:56.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:56.652+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:56.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:56.652+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:56.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:56.653+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:57.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:56.953+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:56.953+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:56.953+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:57.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:56.954+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:57.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:57.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:57.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:57.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:57.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:57.632+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:57.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:57.633+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:57.992 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:36:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:36:57.855+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29696 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:36:57.992 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:57.731+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:57.992 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:57.731+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:57.992 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:57.731+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:57.992 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:57.732+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:57.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:57.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:57.989+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:58.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:57.990+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:59.027 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:58.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:59.027 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:58.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:59.027 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:58.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:36:59.027 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:58.634+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:36:59.027 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:58.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:59.028 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:58.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:59.028 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:58.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:36:59.028 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:58.732+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:36:59.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:59.025+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:59.025+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:59.025+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:36:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:36:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:36:59.026+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:00.002 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:59.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:00.003 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:59.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:00.003 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:59.770+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:00.003 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:36:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:36:59.770+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:00.003 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:59.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:00.003 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:59.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:00.003 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:59.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:00.003 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:36:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:36:59.657+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:00.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:00.000+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:00.000+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:00.000+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:00.001+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:00.977 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:00.793+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:00.977 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:00.793+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:00.977 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:00.793+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:00.977 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:00.794+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:00.978 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:00.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:00.978 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:00.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:00.978 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:00.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:00.978 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:00.627+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:01.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:00.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:00.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:00.975+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:01.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:00.976+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:01.994 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:01.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:01.994 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:01.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:01.995 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:01.747+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:01.995 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:01.748+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:01.995 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:01.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:01.995 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:01.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:01.996 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:01.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:01.996 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:01.647+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:02.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:01.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:01.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:01.992+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:01.993+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:03.043 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:37:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:37:02.859+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29703 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:37:03.043 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:02.786+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:03.043 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:02.786+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:03.044 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:02.786+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:03.044 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:02.787+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:03.044 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:02.692+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:03.044 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:02.692+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:03.044 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:02.692+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:03.044 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:02.693+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:03.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:03.041+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:03.041+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:03.041+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:03.042+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:04.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:03.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:03.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:03.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:03.835+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:03.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:03.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:04.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:03.646+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:04.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:03.646+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:04.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:04.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:04.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:04.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:04.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:04.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:04.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:04.080+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:04.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:04.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:04.809+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:05.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:04.810+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:04.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:05.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:04.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:05.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:04.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:05.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:04.669+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:05.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:05.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:05.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:05.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:05.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:05.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:05.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:05.080+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:05.953 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:05.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:05.953 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:05.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:05.953 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:05.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:05.953 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:05.770+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:05.954 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:05.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:05.954 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:05.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:05.954 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:05.663+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:05.954 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:05.664+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:06.101+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:06.101+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:06.101+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:06.102+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:07.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:06.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:06.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:06.755+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:07.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:06.756+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:06.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:07.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:06.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:07.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:06.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:07.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:06.616+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:07.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:07.130+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:07.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:07.130+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:07.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:07.130+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:07.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:07.131+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:08.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:07.721+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:07.721+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:07.721+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:07.722+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:07.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:08.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:07.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:08.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:07.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:08.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:07.658+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:08.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:37:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:37:07.861+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29711 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:37:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:08.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:08.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:08.162+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:08.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:08.163+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:09.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:08.743+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:08.743+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:08.743+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:08.744+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:08.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:08.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:08.626+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:08.626+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:09.123+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:09.123+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:09.123+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:09.123+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:09.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:09.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:09.732+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:09.732+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:10.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:09.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:10.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:09.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:10.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:09.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:10.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:09.581+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:10.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:10.142+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:10.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:10.142+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:10.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:10.142+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:10.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:10.143+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:11.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:10.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:10.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:10.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:10.589+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:10.686+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:10.686+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:10.686+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:11.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:10.686+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:11.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:11.175+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:11.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:11.175+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:11.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:11.175+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:11.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:11.176+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:12.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:11.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:11.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:11.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:11.590+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:12.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:11.672+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:12.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:11.672+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:12.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:11.672+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:12.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:11.672+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:12.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:12.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:12.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:12.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:12.204+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:12.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:12.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:12.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:12.713+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:13.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:12.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:13.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:12.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:13.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:12.615+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:13.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:12.616+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:13.083 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:37:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:37:12.865+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29718 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:37:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:13.199+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:13.199+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:13.199+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:13.200+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:13.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:13.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:13.759+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:13.761+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:14.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:13.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:14.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:13.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:14.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:13.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:14.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:13.652+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:14.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:14.184+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:14.184+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:14.184+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:14.185+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:14.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:14.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:14.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:14.769+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:15.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:14.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:15.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:14.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:15.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:14.643+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:15.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:14.644+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:15.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:15.155+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:15.155+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:15.155+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:15.156+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:15.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:15.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:15.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:15.730+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:15.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:15.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:15.617+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:15.617+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:16.141+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:16.141+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:16.141+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:16.142+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:17.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:16.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:16.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:16.730+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:16.731+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:17.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:16.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:17.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:16.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:17.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:16.587+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:17.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:16.588+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:17.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:17.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:17.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:17.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:17.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:17.117+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:17.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:17.117+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:17.870 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:17.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:17.871 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:17.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:17.871 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:17.746+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:17.871 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:17.747+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:17.871 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:17.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:17.871 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:17.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:17.872 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:17.605+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:17.872 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:17.606+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:18.130 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:37:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:37:17.869+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29726 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:37:18.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:18.129+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:18.129+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:18.129+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:18.130+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:18.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:18.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:18.718+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:18.719+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:18.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:18.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:18.624+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:18.625+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:19.141+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:19.141+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:19.141+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:19.141+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:20.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:19.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:19.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:19.713+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:19.713+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:19.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:19.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:20.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:19.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:20.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:19.586+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:20.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:20.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:20.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:20.149+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:20.149+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:21.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:20.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:20.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:20.664+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:20.665+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:20.601+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:20.601+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:21.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:20.601+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:21.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:20.602+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:21.511 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:21.150+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:21.511 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:21.150+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:21.511 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:21.150+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:21.512 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:21.151+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:21.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:21.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:21.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:21.660+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:21.661+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:21.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:21.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:21.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:21.638+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:22.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:22.163+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:22.163+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:22.163+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:22.164+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:22.873 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:22.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:22.874 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:22.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:22.874 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:22.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:22.874 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:22.604+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:22.874 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:22.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:22.875 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:22.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:22.875 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:22.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:22.875 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:22.691+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:23.165 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:37:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:37:22.872+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29733 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:37:23.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:23.163+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:23.163+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:23.163+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:23.164+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:23.599+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:23.599+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:23.599+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:23.600+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:24.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:23.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:24.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:23.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:24.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:23.690+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:24.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:23.691+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:24.384 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:24.128+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:24.385 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:24.128+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:24.385 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:24.128+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:24.385 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:24.129+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:24.703+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:24.703+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:24.703+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:24.704+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:25.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:24.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:25.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:24.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:25.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:24.645+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:25.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:24.646+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:25.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:25.109+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:25.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:25.109+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:25.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:25.109+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:25.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:25.110+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:25.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:25.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:26.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:25.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:26.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:25.723+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:26.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:25.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:26.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:25.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:26.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:25.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:26.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:25.630+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:26.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:26.139+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:26.139+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:26.139+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:26.140+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:26.760+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:26.760+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:26.760+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:26.761+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:26.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:27.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:26.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:27.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:26.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:27.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:26.616+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:27.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:27.171+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:27.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:27.171+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:27.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:27.171+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:27.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:27.172+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:27.876 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:27.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:27.877 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:27.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:27.877 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:27.796+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:27.877 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:27.796+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:27.877 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:27.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:27.877 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:27.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:27.877 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:27.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:27.878 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:27.586+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:28.207 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:37:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:37:27.874+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29741 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:37:28.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:28.205+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:28.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:28.205+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:28.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:28.205+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:28.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:28.206+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:28.801+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:28.801+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:28.801+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:29.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:28.802+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:29.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:28.594+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:29.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:28.594+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:29.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:28.594+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:29.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:28.595+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:29.226+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:29.226+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:29.226+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:29.227+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:30.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:29.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:29.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:29.821+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:30.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:29.822+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:30.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:29.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:30.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:29.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:30.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:29.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:30.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:29.604+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:30.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:30.210+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:30.210+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:30.210+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:30.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:30.210+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:30.773+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:30.773+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:30.773+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:30.774+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:30.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:31.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:30.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:31.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:30.637+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:31.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:30.638+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:31.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:31.225+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:31.225+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:31.225+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:31.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:31.225+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:31.754+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:31.754+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:31.754+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:32.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:31.755+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:32.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:31.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:32.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:31.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:32.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:31.669+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:32.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:31.670+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:32.202+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:32.202+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:32.202+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:32.203+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:33.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:37:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:37:32.877+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29748 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:37:33.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:32.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:33.084 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:32.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:33.084 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:32.695+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:33.084 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:32.695+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:33.084 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:32.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:33.084 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:32.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:33.084 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:32.805+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:33.085 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:32.806+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:33.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:33.240+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:33.240+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:33.240+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:33.241+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:34.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:33.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:33.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:33.757+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:33.758+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:33.674+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:33.674+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:34.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:33.674+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:34.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:33.674+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:34.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:34.202+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:34.202+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:34.202+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:34.203+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:35.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:34.668+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:34.668+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:34.668+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:35.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:34.668+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:34.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:34.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:35.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:34.769+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:35.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:34.770+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:35.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:35.184+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:35.184+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:35.184+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:35.185+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:36.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:35.788+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:35.788+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:35.788+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:35.788+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:35.691+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:35.691+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:35.691+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:36.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:35.692+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:36.168+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:36.168+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:36.168+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:36.169+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:36.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:36.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:36.829+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:36.830+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:37.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:36.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:37.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:36.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:37.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:36.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:37.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:36.738+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:37.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:37.139+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:37.139+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:37.139+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:37.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:37.140+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:38.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:37:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:37:37.880+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29756 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:37:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:37.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:37.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:37.842+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:38.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:37.843+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:38.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:37.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:38.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:37.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:38.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:37.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:38.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:37.773+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:38.171+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:38.171+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:38.171+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:38.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:38.172+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:39.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:38.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:38.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:38.874+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:38.874+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:38.802+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:38.802+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:38.802+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:39.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:38.803+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:39.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:39.125+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:39.125+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:39.125+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:39.126+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:40.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:39.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:39.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:39.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:40.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:39.884+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:39.803+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:39.803+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:40.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:39.803+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:40.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:39.804+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:40.150+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:40.150+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:40.150+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:40.151+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:41.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:40.909+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:40.909+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:40.909+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:41.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:40.909+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:40.789+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:40.789+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:41.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:40.789+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:41.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:40.789+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:41.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:41.135+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:41.135+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:41.135+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:41.136+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:41.743+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:41.743+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:41.743+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:41.744+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:42.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:41.942+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:42.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:41.942+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:42.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:41.942+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:42.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:41.943+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:42.163+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:42.163+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:42.163+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:42.163+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:43.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:37:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:37:42.882+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29763 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:37:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:42.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:42.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:42.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:43.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:42.954+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:43.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:42.784+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:43.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:42.784+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:43.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:42.784+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:43.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:42.785+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:43.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:43.202+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:43.202+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:43.202+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:43.202+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:44.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:43.937+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:43.937+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:43.937+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:43.938+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:43.798+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:43.798+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:43.798+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:43.799+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:44.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:44.206+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:44.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:44.206+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:44.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:44.206+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:44.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:44.206+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:44.892 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:44.833+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:44.893 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:44.833+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:44.893 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:44.833+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:44.893 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:44.834+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:45.185 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:44.890+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:45.185 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:44.890+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:45.185 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:44.890+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:45.186 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:44.890+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:45.186 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:45.182+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:45.186 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:45.182+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:45.186 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:45.182+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:45.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:45.183+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:46.202 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:45.845+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:46.202 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:45.845+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:46.202 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:45.845+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:46.202 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:45.845+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:46.202 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:45.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:46.203 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:45.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:46.203 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:45.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:46.203 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:45.931+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:46.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:46.199+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:46.199+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:46.199+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:46.200+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:47.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:46.827+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:46.827+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:46.827+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:46.828+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:46.980+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:46.980+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:47.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:46.980+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:47.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:46.981+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:47.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:47.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:47.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:47.242+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:47.242+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:48.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:37:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:37:47.886+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29771 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:37:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:47.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:47.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:47.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:47.815+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:48.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:47.983+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:48.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:47.983+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:48.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:47.983+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:48.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:47.984+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:48.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:48.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:48.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:48.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:48.284+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:48.807+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:48.807+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:48.807+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:48.807+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:49.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:48.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:49.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:48.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:49.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:48.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:49.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:48.935+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:49.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:49.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:49.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:49.280+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:49.281+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:50.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:49.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:49.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:49.894+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:50.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:49.895+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:49.780+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:49.780+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:50.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:49.780+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:50.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:49.781+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:50.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:50.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:50.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:50.315+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:50.316+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:50.936+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:50.936+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:50.936+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:51.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:50.936+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:51.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:50.776+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:51.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:50.776+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:51.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:50.776+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:51.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:50.777+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:51.802 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:51.344+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:51.803 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:51.344+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:51.803 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:51.344+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:51.803 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:51.345+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:52.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:51.954+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:51.954+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:51.954+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:52.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:51.955+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:52.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:51.801+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:52.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:51.801+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:52.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:51.801+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:52.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:51.801+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:52.386+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:52.386+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:52.386+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:52.387+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:53.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:37:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:37:52.889+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29778 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:37:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:52.994+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:52.994+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:52.994+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:52.995+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:53.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:52.838+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:53.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:52.838+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:53.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:52.838+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:53.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:52.839+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:53.738 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:53.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:53.739 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:53.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:53.739 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:53.404+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:53.739 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:53.405+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:54.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:53.810+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:54.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:53.810+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:54.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:53.810+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:54.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:53.811+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:53.952+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:53.952+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:54.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:53.952+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:54.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:53.953+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:54.823 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:54.366+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:54.824 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:54.366+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:54.824 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:54.366+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:54.824 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:54.367+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:55.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:54.913+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:55.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:54.913+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:55.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:54.913+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:55.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:54.914+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:55.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:54.822+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:55.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:54.822+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:55.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:54.822+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:55.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:54.823+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:55.354 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:55.352+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:55.355 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:55.352+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:55.355 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:55.352+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:55.355 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:55.353+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:55.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:55.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:55.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:55.931+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:56.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:55.858+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:56.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:55.858+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:56.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:55.858+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:56.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:55.859+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:56.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:56.352+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:56.352+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:56.352+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:56.352+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:57.203 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:56.938+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:57.203 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:56.938+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:57.203 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:56.938+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:57.203 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:56.938+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:57.204 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:56.866+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:57.204 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:56.866+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:57.204 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:56.866+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:57.204 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:56.867+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:57.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:57.341+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:57.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:57.341+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:57.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:57.341+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:57.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:57.342+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:58.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:37:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:37:57.892+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29786 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:37:58.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:57.895+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:58.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:57.895+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:58.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:57.895+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:58.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:57.896+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:57.961+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:58.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:57.961+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:58.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:57.961+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:58.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:57.961+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:58.330+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:58.330+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:58.330+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:58.331+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:59.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:58.881+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:59.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:58.881+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:59.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:58.881+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:37:59.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:58.882+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:37:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:58.967+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:58.967+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:59.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:58.967+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:37:59.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:37:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:37:58.968+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:37:59.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:59.354+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:59.354+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:59.354+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:37:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:37:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:37:59.355+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:00.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:59.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:00.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:59.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:00.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:59.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:00.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:37:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:37:59.874+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:00.011+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:00.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:00.011+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:00.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:00.011+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:00.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:00.011+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:00.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:00.333+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:00.334+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:00.334+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:00.334+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:01.313 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:00.971+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:01.313 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:00.971+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:01.314 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:00.971+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:01.314 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:00.971+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:01.314 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:00.919+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:01.314 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:00.919+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:01.314 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:00.919+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:01.314 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:00.920+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:01.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:01.311+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:01.311+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:01.311+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:01.312+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:02.272 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:01.894+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:02.272 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:01.894+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:02.273 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:01.894+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:02.273 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:01.895+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:02.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:01.961+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:02.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:01.961+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:02.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:01.961+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:02.274 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:01.962+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:02.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:02.270+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:02.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:02.270+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:02.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:02.270+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:02.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:02.271+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:03.304 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:02.964+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:03.304 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:02.964+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:03.305 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:02.964+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:03.305 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:02.964+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:03.305 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:02.846+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:03.305 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:02.846+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:03.305 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:02.846+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:03.305 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:02.847+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:03.306 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:38:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:38:02.895+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29793 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:38:03.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:03.302+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:03.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:03.302+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:03.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:03.302+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:03.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:03.303+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:03.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:03.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:03.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:04.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:03.933+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:03.897+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:04.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:03.897+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:04.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:03.897+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:04.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:03.897+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:04.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:04.301+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:04.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:04.301+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:04.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:04.301+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:04.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:04.302+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:04.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:04.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:04.930+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:04.931+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:04.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:04.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:05.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:04.874+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:05.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:04.875+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:05.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:05.334+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:05.334+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:05.334+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:05.335+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:06.294 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:05.869+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:06.294 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:05.870+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:06.294 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:05.870+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:06.294 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:05.870+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:06.295 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:05.920+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:06.295 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:05.920+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:06.295 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:05.920+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:06.295 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:05.921+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:06.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:06.292+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:06.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:06.292+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:06.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:06.292+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:06.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:06.293+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:07.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:06.829+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:07.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:06.829+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:07.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:06.829+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:07.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:06.830+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:06.969+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:06.969+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:06.969+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:07.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:06.969+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:07.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:07.328+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:07.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:07.328+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:07.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:07.328+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:07.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:07.329+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:08.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:38:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:38:07.897+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29801 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:38:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:07.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:07.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:07.935+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:07.936+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:08.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:07.808+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:08.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:07.808+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:08.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:07.808+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:08.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:07.808+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:08.826 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:08.346+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:08.827 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:08.346+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:08.827 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:08.346+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:08.827 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:08.346+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:09.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:08.962+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:08.962+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:08.962+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:09.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:08.963+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:08.825+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:08.825+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:08.825+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:09.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:08.825+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:09.312+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:09.312+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:09.312+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:09.313+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:09.962+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:09.962+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:09.962+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:10.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:09.963+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:09.858+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:10.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:09.858+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:10.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:09.858+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:10.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:09.859+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:10.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:10.334+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:10.334+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:10.334+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:10.335+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:11.325 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:10.967+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:11.325 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:10.967+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:11.325 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:10.967+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:11.326 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:10.967+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:11.326 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:10.864+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:11.326 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:10.864+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:11.326 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:10.864+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:11.326 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:10.865+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:11.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:11.323+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:11.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:11.323+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:11.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:11.323+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:11.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:11.324+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:12.289 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:11.983+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:12.289 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:11.983+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:12.289 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:11.983+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:12.290 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:11.983+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:12.290 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:11.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:12.290 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:11.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:12.290 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:11.862+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:12.290 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:11.863+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:12.287+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:12.287+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:12.287+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:12.288+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:13.080 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:38:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:38:12.900+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29808 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:38:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:12.974+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:12.974+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:12.974+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:12.975+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:13.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:12.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:13.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:12.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:13.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:12.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:13.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:12.815+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:13.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:13.259+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:13.259+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:13.259+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:13.260+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:14.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:13.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:13.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:13.933+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:13.933+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:14.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:13.827+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:14.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:13.827+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:14.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:13.827+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:14.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:13.828+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:14.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:14.271+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:14.271+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:14.271+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:14.271+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:14.917 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:14.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:14.917 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:14.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:14.917 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:14.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:14.918 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:14.815+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:15.254 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:14.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:15.254 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:14.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:15.254 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:14.915+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:15.254 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:14.916+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:15.252+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:15.252+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:15.252+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:15.253+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:15.943+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:15.943+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:15.943+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:15.944+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:16.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:15.780+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:15.780+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:15.780+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:15.781+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:16.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:16.222+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:16.222+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:16.222+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:16.223+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:16.968 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:16.757+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:16.968 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:16.757+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:16.968 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:16.757+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:16.968 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:16.758+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:17.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:16.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:16.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:16.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:16.967+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:17.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:17.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:17.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:17.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:17.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:17.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:17.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:17.197+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:17.905 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:17.789+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:17.905 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:17.789+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:17.905 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:17.789+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:17.905 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:17.789+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:18.228 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:38:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:38:17.903+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29816 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:38:18.229 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:17.937+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:18.229 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:17.937+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:18.229 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:17.937+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:18.230 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:17.938+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:18.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:18.227+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:18.227+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:18.227+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:18.227+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:19.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:18.977+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:18.977+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:18.977+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:19.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:18.978+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:18.812+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:18.812+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:18.812+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:18.812+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:19.191+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:19.191+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:19.191+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:19.192+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:19.996+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:19.996+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:19.996+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:20.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:19.996+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:20.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:19.805+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:20.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:19.805+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:20.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:19.805+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:20.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:19.806+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:20.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:20.167+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:20.167+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:20.167+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:20.168+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:21.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:21.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:21.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:21.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:21.005+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:21.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:20.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:21.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:20.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:21.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:20.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:21.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:20.794+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:21.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:21.119+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:21.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:21.119+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:21.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:21.119+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:21.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:21.120+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:22.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:21.802+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:21.802+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:21.802+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:21.802+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:22.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:22.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:22.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:22.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:22.039+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:22.094+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:22.094+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:22.094+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:22.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:22.095+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:23.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:22.840+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:23.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:22.840+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:23.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:22.840+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:23.067 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:22.841+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:23.067 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:38:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:38:22.905+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29823 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:38:23.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:23.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:23.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:23.063+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:23.064+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:23.068+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:23.068+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:23.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:23.068+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:23.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:23.069+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:24.040 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:23.886+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:24.040 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:23.886+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:24.040 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:23.886+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:24.041 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:23.886+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:24.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:24.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:24.079+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:24.080+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:24.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:24.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:24.037+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:24.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:24.038+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:25.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:24.905+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:25.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:24.905+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:25.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:24.905+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:25.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:24.905+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:25.024+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:25.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:25.024+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:25.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:25.024+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:25.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:25.024+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:25.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:25.126+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:25.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:25.126+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:25.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:25.126+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:25.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:25.126+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:26.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:26.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:26.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:26.041+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:25.885+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:26.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:25.885+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:26.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:25.885+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:26.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:25.886+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:26.135+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:26.135+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:26.135+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:26.136+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:27.103 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:27.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:27.103 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:27.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:27.104 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:27.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:27.104 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:27.044+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:27.104 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:26.849+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:27.104 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:26.849+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:27.104 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:26.849+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:27.104 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:26.850+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:27.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:27.101+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:27.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:27.101+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:27.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:27.101+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:27.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:27.102+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:28.081 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:38:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:38:27.909+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29831 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:38:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:28.023+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:28.023+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:28.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:28.023+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:28.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:28.024+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:28.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:27.890+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:28.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:27.890+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:28.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:27.890+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:28.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:27.890+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:28.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:28.115+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:28.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:28.115+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:28.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:28.115+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:28.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:28.115+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:29.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:28.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:28.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:28.981+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:28.981+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:29.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:28.940+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:29.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:28.940+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:29.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:28.940+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:29.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:28.941+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:29.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:29.159+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:29.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:29.159+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:29.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:29.159+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:29.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:29.160+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:30.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:30.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:30.021+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:30.021+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:30.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:29.984+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:30.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:29.984+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:30.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:29.984+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:30.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:29.985+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:30.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:30.184+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:30.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:30.184+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:30.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:30.184+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:30.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:30.184+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:31.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:31.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:31.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:31.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:31.007+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:31.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:31.033+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:31.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:31.033+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:31.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:31.033+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:31.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:31.033+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:31.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:31.221+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:31.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:31.221+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:31.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:31.221+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:31.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:31.221+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:32.056+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:32.056+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:32.056+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:32.057+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:32.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:32.083+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:32.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:32.083+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:32.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:32.083+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:32.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:32.084+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:32.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:32.256+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:32.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:32.256+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:32.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:32.256+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:32.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:32.257+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:33.284 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:33.059+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:33.285 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:33.059+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:33.285 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:33.059+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:33.285 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:33.060+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:33.285 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:33.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:33.285 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:33.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:33.286 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:33.038+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:33.286 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:33.039+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:33.286 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:38:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:38:32.912+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29838 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:38:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:33.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:33.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:33.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:33.284+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:34.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:34.106+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:34.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:34.106+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:34.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:34.106+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:34.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:34.107+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:34.030+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:34.030+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:34.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:34.030+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:34.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:34.030+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:34.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:34.269+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:34.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:34.269+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:34.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:34.269+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:34.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:34.270+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:35.279 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:35.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:35.279 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:35.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:35.279 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:35.017+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:35.280 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:35.017+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:35.280 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:35.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:35.280 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:35.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:35.280 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:35.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:35.280 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:35.098+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:35.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:35.277+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:35.277+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:35.277+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:35.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:35.278+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:36.329 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:36.111+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:36.329 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:36.111+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:36.329 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:36.111+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:36.329 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:36.112+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:36.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:36.045+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:36.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:36.045+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:36.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:36.045+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:36.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:36.046+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:36.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:36.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:36.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:36.327+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:36.328+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:37.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:37.150+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:37.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:37.150+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:37.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:37.150+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:37.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:37.150+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:37.068+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:37.068+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:37.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:37.068+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:37.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:37.069+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:37.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:37.286+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:37.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:37.286+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:37.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:37.286+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:37.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:37.286+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:38.252 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:38:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:38:37.915+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29846 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:38:38.253 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:38.044+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:38.253 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:38.044+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:38.253 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:38.044+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:38.253 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:38.044+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:38.254 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:38.149+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:38.254 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:38.149+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:38.254 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:38.149+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:38.254 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:38.150+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:38.254 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:38.251+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:38.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:38.251+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:38.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:38.251+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:38.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:38.252+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:39.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:39.177+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:39.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:39.177+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:39.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:39.177+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:39.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:39.178+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:39.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:39.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:39.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:39.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:39.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:39.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:39.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:39.090+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:39.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:39.241+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:39.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:39.241+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:39.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:39.241+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:39.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:39.242+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:40.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:40.147+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:40.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:40.148+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:40.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:40.148+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:40.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:40.148+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:40.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:40.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:40.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:40.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:40.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:40.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:40.128+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:40.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:40.279+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:40.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:40.279+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:40.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:40.279+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:40.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:40.279+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:41.326 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:41.198+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:41.326 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:41.198+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:41.326 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:41.198+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:41.326 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:41.198+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:41.326 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:41.087+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:41.327 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:41.087+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:41.327 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:41.087+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:41.327 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:41.088+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:41.327 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:41.322+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:41.327 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:41.322+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:41.327 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:41.322+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:41.327 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:41.323+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:42.352 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:42.246+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:42.353 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:42.246+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:42.353 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:42.246+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:42.353 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:42.247+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:42.353 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:42.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:42.353 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:42.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:42.353 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:42.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:42.354 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:42.095+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:42.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:42.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:42.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:42.351+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:42.351+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:43.203 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:38:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:38:42.918+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29853 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:38:43.204 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:43.100+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:43.204 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:43.100+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:43.204 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:43.100+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:43.204 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:43.101+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:43.391+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:43.391+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:43.391+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:43.392+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:43.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:43.202+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:43.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:43.202+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:43.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:43.202+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:43.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:43.203+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:44.357 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:44.174+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:44.357 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:44.174+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:44.357 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:44.174+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:44.358 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:44.175+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:44.358 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:44.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:44.358 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:44.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:44.358 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:44.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:44.358 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:44.095+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:44.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:44.355+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:44.355+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:44.355+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:44.356+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:45.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:45.082+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:45.082+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:45.082+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:45.083+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:45.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:45.185+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:45.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:45.185+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:45.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:45.185+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:45.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:45.185+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:45.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:45.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:45.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:45.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:45.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:45.307+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:45.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:45.307+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:46.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:46.118+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:46.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:46.118+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:46.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:46.118+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:46.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:46.119+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:46.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:46.234+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:46.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:46.234+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:46.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:46.234+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:46.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:46.235+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:46.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:46.292+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:46.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:46.292+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:46.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:46.292+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:46.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:46.293+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:47.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:47.107+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:47.107+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:47.107+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:47.108+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:47.210+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:47.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:47.210+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:47.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:47.210+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:47.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:47.211+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:47.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:47.270+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:47.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:47.270+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:47.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:47.270+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:47.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:47.271+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:48.232 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:38:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:38:47.921+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29861 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:38:48.233 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:48.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:48.233 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:48.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:48.233 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:48.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:48.233 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:48.158+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:48.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:48.247+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:48.247+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:48.247+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:48.248+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:48.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:48.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:48.260+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:48.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:48.260+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:49.225+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:49.225+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:49.225+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:49.226+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:49.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:49.132+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:49.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:49.132+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:49.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:49.132+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:49.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:49.133+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:49.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:49.241+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:49.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:49.241+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:49.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:49.241+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:49.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:49.241+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:50.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:50.214+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:50.214+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:50.214+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:50.215+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:50.091+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:50.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:50.091+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:50.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:50.091+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:50.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:50.092+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:50.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:50.199+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:50.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:50.199+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:50.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:50.199+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:50.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:50.200+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:51.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:51.243+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:51.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:51.243+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:51.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:51.243+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:51.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:51.244+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:51.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:51.130+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:51.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:51.130+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:51.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:51.130+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:51.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:51.130+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:51.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:51.196+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:51.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:51.196+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:51.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:51.196+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:51.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:51.196+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:52.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:52.234+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:52.234+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:52.234+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:52.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:52.235+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:52.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:52.126+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:52.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:52.126+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:52.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:52.126+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:52.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:52.127+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:52.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:52.289+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:52.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:52.289+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:52.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:52.289+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:52.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:52.290+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:53.269 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:38:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:38:52.924+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29868 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:38:53.269 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:53.123+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:53.270 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:53.123+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:53.270 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:53.123+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:53.270 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:53.124+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:53.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:53.267+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:53.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:53.267+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:53.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:53.267+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:53.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:53.268+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:53.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:53.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:53.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:53.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:53.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:53.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:53.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:53.284+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:54.222+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:54.222+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:54.222+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:54.223+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:54.114+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:54.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:54.114+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:54.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:54.114+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:54.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:54.115+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:54.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:54.251+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:54.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:54.251+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:54.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:54.251+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:54.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:54.251+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:55.239 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:55.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:55.239 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:55.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:55.239 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:55.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:55.240 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:55.090+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:55.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:55.245+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:55.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:55.245+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:55.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:55.245+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:55.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:55.246+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:55.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:55.237+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:55.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:55.237+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:55.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:55.237+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:55.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:55.238+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:56.073+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:56.073+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:56.073+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:56.074+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:56.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:56.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:56.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:56.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:56.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:56.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:56.224+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:56.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:56.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:56.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:56.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:56.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:56.239+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:56.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:56.239+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:57.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:57.090+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:57.090+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:57.090+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:57.091+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:57.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:57.224+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:57.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:57.224+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:57.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:57.224+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:57.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:57.225+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:57.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:57.216+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:57.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:57.216+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:57.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:57.216+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:57.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:57.217+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:58.184 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:58.116+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:58.185 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:58.116+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:58.185 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:58.116+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:58.185 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:58.116+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:58.185 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:38:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:38:57.926+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29876 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:38:58.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:58.205+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:58.205+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:58.205+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:58.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:58.206+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:58.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:58.181+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:58.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:58.181+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:58.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:58.181+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:58.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:58.182+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:38:59.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:59.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:59.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:59.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:38:59.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:38:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:38:59.120+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:59.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:59.248+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:59.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:59.248+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:59.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:59.248+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:38:59.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:38:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:38:59.249+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:38:59.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:59.197+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:59.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:59.197+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:59.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:59.197+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:38:59.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:38:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:38:59.198+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:00.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:00.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:00.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:00.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:00.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:00.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:00.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:00.122+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:00.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:00.272+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:00.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:00.272+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:00.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:00.272+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:00.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:00.273+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:00.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:00.206+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:00.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:00.206+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:00.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:00.206+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:00.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:00.206+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:01.305+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:01.305+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:01.305+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:01.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:01.306+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:01.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:01.157+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:01.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:01.157+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:01.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:01.157+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:01.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:01.158+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:01.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:01.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:01.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:01.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:01.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:01.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:01.583 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:01.106+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:02.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:02.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:02.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:02.124+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:02.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:02.337+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:02.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:02.337+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:02.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:02.337+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:02.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:02.338+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:02.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:02.207+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:02.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:02.207+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:02.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:02.207+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:02.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:02.207+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:03.211 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:39:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:39:02.930+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29883 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:39:03.211 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:03.146+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:03.212 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:03.146+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:03.212 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:03.146+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:03.212 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:03.147+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:03.485 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:03.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:03.485 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:03.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:03.486 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:03.353+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:03.486 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:03.354+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:03.486 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:03.209+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:03.486 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:03.209+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:03.486 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:03.209+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:03.486 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:03.210+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:04.380 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:04.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:04.380 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:04.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:04.381 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:04.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:04.381 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:04.128+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:04.381 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:04.194+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:04.381 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:04.194+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:04.381 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:04.194+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:04.381 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:04.195+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:04.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:04.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:04.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:04.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:04.379+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:05.418 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:05.117+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:05.418 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:05.117+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:05.419 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:05.117+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:05.419 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:05.117+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:05.419 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:05.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:05.419 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:05.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:05.419 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:05.221+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:05.419 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:05.222+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:05.761 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:05.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:05.761 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:05.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:05.761 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:05.416+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:05.761 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:05.417+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:06.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:06.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:06.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:06.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:06.147+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:06.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:06.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:06.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:06.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:06.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:06.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:06.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:06.393+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:06.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:06.253+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:06.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:06.253+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:06.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:06.253+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:06.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:06.254+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:07.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:07.423+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:07.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:07.423+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:07.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:07.423+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:07.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:07.423+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:07.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:07.192+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:07.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:07.192+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:07.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:07.192+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:07.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:07.193+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:07.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:07.205+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:07.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:07.205+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:07.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:07.205+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:07.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:07.206+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:08.244 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:39:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:39:07.932+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29891 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:39:08.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:08.252+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:08.252+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:08.252+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:08.253+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:08.242+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:08.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:08.242+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:08.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:08.242+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:08.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:08.243+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:08.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:08.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:08.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:08.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:08.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:08.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:08.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:08.428+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:09.197+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:09.197+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:09.197+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:09.197+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:09.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:09.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:09.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:09.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:09.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:09.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:09.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:09.384+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:09.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:09.224+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:09.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:09.224+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:09.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:09.224+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:09.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:09.225+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:10.244+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:10.244+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:10.244+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:10.245+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:10.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:10.336+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:10.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:10.336+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:10.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:10.336+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:10.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:10.336+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:10.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:10.214+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:10.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:10.214+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:10.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:10.214+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:10.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:10.215+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:11.232+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:11.232+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:11.232+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:11.233+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:11.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:11.356+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:11.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:11.356+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:11.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:11.356+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:11.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:11.357+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:11.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:11.208+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:11.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:11.208+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:11.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:11.208+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:11.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:11.209+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:12.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:12.245+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:12.245+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:12.245+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:12.246+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:12.183+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:12.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:12.183+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:12.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:12.183+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:12.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:12.184+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:12.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:12.350+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:12.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:12.350+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:12.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:12.350+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:12.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:12.351+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:13.259 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:39:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:39:12.935+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29898 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:39:13.259 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:13.166+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:13.260 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:13.166+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:13.260 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:13.166+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:13.260 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:13.167+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:13.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:13.350+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:13.350+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:13.350+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:13.351+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:13.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:13.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:13.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:13.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:13.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:13.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:13.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:13.258+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:14.269+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:14.269+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:14.269+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:14.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:14.270+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:14.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:14.364+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:14.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:14.364+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:14.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:14.364+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:14.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:14.365+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:14.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:14.144+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:14.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:14.144+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:14.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:14.144+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:14.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:14.145+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:15.297 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:15.139+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:15.309 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:15.139+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:15.309 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:15.139+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:15.310 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:15.140+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:15.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:15.408+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:15.408+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:15.408+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:15.409+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:15.295+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:15.295+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:15.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:15.295+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:15.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:15.296+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:16.279+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:16.279+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:16.279+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:16.279+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:16.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:16.363+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:16.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:16.363+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:16.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:16.363+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:16.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:16.364+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:16.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:16.176+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:16.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:16.176+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:16.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:16.176+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:16.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:16.177+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:17.280+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:17.280+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:17.280+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:17.280+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:17.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:17.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:17.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:17.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:17.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:17.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:17.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:17.401+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:17.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:17.204+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:17.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:17.204+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:17.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:17.204+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:17.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:17.204+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:18.233 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:39:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:39:17.938+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29906 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:39:18.234 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:18.187+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:18.234 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:18.187+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:18.234 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:18.187+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:18.234 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:18.188+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:18.232+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:18.232+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:18.232+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:18.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:18.233+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:18.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:18.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:18.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:18.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:18.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:18.383+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:18.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:18.384+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:19.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:19.396+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:19.396+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:19.396+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:19.397+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:19.264+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:19.264+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:19.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:19.264+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:19.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:19.264+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:19.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:19.211+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:19.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:19.211+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:19.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:19.211+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:19.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:19.212+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:20.447 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:20.193+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:20.448 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:20.193+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:20.448 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:20.193+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:20.448 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:20.194+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:20.448 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:20.239+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:20.448 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:20.239+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:20.448 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:20.239+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:20.448 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:20.240+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:20.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:20.446+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:20.446+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:20.446+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:20.446+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:21.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:21.226+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:21.226+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:21.226+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:21.227+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:21.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:21.405+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:21.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:21.405+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:21.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:21.405+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:21.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:21.406+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:21.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:21.237+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:21.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:21.237+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:21.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:21.237+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:21.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:21.237+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:22.432 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:22.177+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:22.433 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:22.177+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:22.433 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:22.177+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:22.433 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:22.178+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:22.433 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:22.256+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:22.433 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:22.256+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:22.434 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:22.256+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:22.434 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:22.257+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:22.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:22.431+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:22.431+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:22.431+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:22.431+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:23.218 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:39:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:39:22.941+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29913 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:39:23.218 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:23.131+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:23.219 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:23.131+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:23.219 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:23.131+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:23.219 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:23.131+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:23.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:23.391+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:23.391+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:23.391+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:23.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:23.391+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:23.216+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:23.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:23.216+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:23.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:23.216+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:23.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:23.217+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:24.438 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:24.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:24.439 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:24.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:24.439 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:24.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:24.439 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:24.121+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:24.439 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:24.193+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:24.439 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:24.193+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:24.439 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:24.193+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:24.440 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:24.194+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:24.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:24.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:24.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:24.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:24.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:24.437+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:25.448 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:25.139+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:25.449 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:25.139+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:25.449 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:25.139+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:25.449 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:25.141+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:25.449 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:25.211+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:25.449 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:25.211+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:25.449 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:25.211+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:25.450 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:25.212+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:25.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:25.446+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:25.446+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:25.446+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:25.447+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:26.401 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:26.105+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:26.401 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:26.105+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:26.401 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:26.105+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:26.401 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:26.105+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:26.401 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:26.164+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:26.402 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:26.164+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:26.402 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:26.164+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:26.402 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:26.165+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:26.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:26.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:26.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:26.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:26.400+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:27.356 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:27.090+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:27.357 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:27.090+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:27.357 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:27.090+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:27.357 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:27.091+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:27.357 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:27.169+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:27.357 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:27.169+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:27.357 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:27.169+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:27.358 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:27.170+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:27.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:27.354+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:27.354+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:27.354+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:27.355+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:28.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:39:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:39:27.944+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29921 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:39:28.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:28.174+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:28.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:28.174+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:28.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:28.174+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:28.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:28.174+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:28.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:28.112+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:28.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:28.112+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:28.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:28.112+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:28.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:28.113+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:28.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:28.346+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:28.346+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:28.346+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:28.347+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:29.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:29.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:29.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:29.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:29.156+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:29.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:29.130+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:29.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:29.130+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:29.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:29.130+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:29.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:29.131+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:29.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:29.346+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:29.346+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:29.346+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:29.347+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:30.394 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:30.119+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:30.395 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:30.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:30.395 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:30.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:30.395 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:30.120+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:30.395 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:30.122+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:30.395 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:30.122+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:30.395 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:30.122+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:30.395 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:30.123+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:30.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:30.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:30.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:30.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:30.393+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:31.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:31.164+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:31.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:31.164+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:31.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:31.164+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:31.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:31.165+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:31.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:31.135+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:31.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:31.135+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:31.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:31.135+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:31.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:31.135+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:31.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:31.355+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:31.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:31.355+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:31.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:31.355+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:31.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:31.356+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:32.178+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:32.178+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:32.178+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:32.179+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:32.144+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:32.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:32.144+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:32.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:32.144+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:32.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:32.145+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:32.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:32.316+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:32.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:32.316+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:32.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:32.316+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:32.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:32.317+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:33.326 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:39:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:39:32.947+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29928 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:39:33.326 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:33.141+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:33.326 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:33.141+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:33.326 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:33.141+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:33.326 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:33.141+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:33.327 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:33.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:33.327 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:33.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:33.327 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:33.098+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:33.327 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:33.098+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:33.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:33.324+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:33.324+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:33.324+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:33.325+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:34.068+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:34.068+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:34.068+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:34.068+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:34.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:34.162+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:34.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:34.162+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:34.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:34.162+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:34.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:34.163+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:34.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:34.338+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:34.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:34.338+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:34.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:34.338+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:34.599 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:34.339+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:35.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:35.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:35.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:35.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:35.065+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:35.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:35.207+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:35.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:35.207+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:35.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:35.207+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:35.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:35.208+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:35.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:35.345+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:35.345+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:35.345+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:35.346+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:36.379 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:36.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:36.380 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:36.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:36.380 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:36.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:36.380 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:36.089+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:36.380 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:36.161+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:36.380 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:36.161+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:36.380 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:36.161+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:36.381 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:36.162+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:36.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:36.377+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:36.377+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:36.377+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:36.377+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:37.041+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:37.041+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:37.041+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:37.042+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:37.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:37.156+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:37.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:37.156+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:37.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:37.156+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:37.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:37.157+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:37.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:37.390+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:37.390+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:37.390+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:37.391+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:38.209 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:38.011+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:38.209 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:38.011+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:38.209 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:38.011+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:38.209 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:38.011+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:38.209 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:39:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:39:37.949+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29936 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:39:38.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:38.206+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:38.206+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:38.206+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:38.207+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:38.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:38.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:38.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:38.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:38.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:38.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:38.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:38.418+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:39.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:39.222+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:39.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:39.222+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:39.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:39.222+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:39.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:39.222+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:39.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:39.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:39.006+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:39.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:39.007+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:39.384+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:39.384+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:39.384+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:39.385+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:40.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:40.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:40.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:40.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:40.273 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:40.018+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:40.274 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:40.019+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:40.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:40.408+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:40.408+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:40.408+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:40.409+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:40.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:40.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:40.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:40.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:40.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:40.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:40.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:40.272+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:41.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:40.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:40.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:40.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:40.996+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:41.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:41.240+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:41.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:41.240+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:41.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:41.240+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:41.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:41.241+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:41.372+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:41.372+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:41.372+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:41.373+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:42.291 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:41.969+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:42.292 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:41.969+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:42.292 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:41.969+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:42.292 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:41.970+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:42.290+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:42.290+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:42.290+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:42.291+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:42.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:42.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:42.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:42.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:42.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:42.403+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:43.298 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:39:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:39:42.953+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29943 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:39:43.298 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:43.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:43.298 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:43.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:43.298 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:43.007+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:43.298 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:43.008+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:43.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:43.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:43.378+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:43.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:43.379+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:43.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:43.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:43.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:43.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:43.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:43.296+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:43.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:43.297+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:44.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:44.256+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:44.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:44.256+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:44.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:44.256+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:44.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:44.256+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:44.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:44.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:44.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:44.043+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:44.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:44.043+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:44.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:44.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:44.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:44.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:44.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:44.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:44.598 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:44.412+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:45.066+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:45.066+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:45.066+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:45.066+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:45.285+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:45.285+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:45.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:45.285+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:45.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:45.285+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:45.696 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:45.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:45.697 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:45.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:45.697 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:45.436+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:45.697 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:45.436+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:46.404 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:46.318+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:46.405 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:46.318+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:46.405 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:46.318+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:46.405 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:46.319+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:46.405 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:46.115+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:46.405 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:46.115+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:46.405 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:46.115+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:46.406 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:46.116+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:46.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:46.403+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:46.403+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:46.403+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:46.403+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:47.153+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:47.153+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:47.153+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:47.153+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:47.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:47.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:47.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:47.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:47.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:47.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:47.272+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:47.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:47.375+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:47.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:47.375+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:47.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:47.375+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:47.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:47.375+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:48.302 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:48.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:48.302 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:48.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:48.303 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:48.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:48.303 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:48.167+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:48.303 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:39:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:39:47.956+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29951 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:39:48.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:48.300+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:48.300+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:48.300+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:48.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:48.301+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:48.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:48.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:48.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:48.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:48.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:48.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:48.390+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:49.180+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:49.180+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:49.180+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:49.181+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:49.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:49.344+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:49.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:49.344+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:49.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:49.344+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:49.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:49.345+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:49.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:49.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:49.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:49.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:49.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:49.418+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:49.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:49.419+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:50.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:50.159+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:50.159+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:50.159+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:50.160+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:50.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:50.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:50.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:50.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:50.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:50.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:50.403+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:50.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:50.368+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:50.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:50.368+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:50.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:50.368+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:50.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:50.369+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:51.403 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:51.134+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:51.404 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:51.134+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:51.404 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:51.134+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:51.404 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:51.134+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:51.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:51.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:51.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:51.402+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:51.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:51.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:51.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:51.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:51.418+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:51.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:51.418+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:52.415 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:52.132+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:52.416 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:52.132+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:52.416 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:52.132+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:52.416 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:52.133+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:52.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:52.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:52.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:52.427+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:52.428+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:52.414+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:52.414+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:52.414+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:52.414+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:53.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:39:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:39:52.959+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29958 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:39:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:53.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:53.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:53.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:53.143+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:53.747 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:53.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:53.747 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:53.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:53.747 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:53.384+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:53.747 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:53.385+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:53.748 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:53.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:53.748 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:53.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:53.748 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:53.399+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:53.748 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:53.399+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:54.519 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:54.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:54.519 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:54.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:54.519 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:54.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:54.519 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:54.182+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:54.520 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:54.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:54.520 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:54.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:54.520 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:54.371+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:54.520 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:54.372+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:54.520 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:54.424+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:54.521 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:54.424+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:54.521 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:54.424+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:54.521 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:54.425+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:55.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:55.431+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:55.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:55.431+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:55.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:55.431+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:55.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:55.431+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:55.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:55.195+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:55.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:55.195+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:55.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:55.195+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:55.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:55.196+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:55.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:55.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:55.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:55.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:55.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:55.400+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:55.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:55.402+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:56.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:56.203+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:56.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:56.203+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:56.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:56.203+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:56.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:56.204+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:56.440+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:56.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:56.440+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:56.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:56.440+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:56.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:56.441+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:56.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:56.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:56.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:56.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:56.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:56.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:56.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:56.392+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:57.444 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:57.180+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:57.444 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:57.180+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:57.444 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:57.180+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:57.444 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:57.181+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:57.445 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:57.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:57.445 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:57.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:57.445 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:57.420+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:57.445 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:57.420+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:57.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:57.442+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:57.442+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:57.442+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:57.443+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:58.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:39:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:39:57.962+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29966 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:39:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:58.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:58.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:58.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:58.143+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:58.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:58.433+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:58.433+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:58.433+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:58.434+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:58.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:58.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:58.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:58.419+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:58.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:58.420+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:59.417 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:59.139+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:59.418 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:59.139+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:59.418 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:59.139+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:39:59.418 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:39:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:39:59.140+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:39:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:59.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:59.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:59.416+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:39:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:39:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:39:59.417+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:39:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:59.438+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:59.438+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:59.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:59.438+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:39:59.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:39:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:39:59.439+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:00.453 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:00.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:00.453 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:00.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:00.453 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:00.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:00.453 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:00.150+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:00.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:00.455+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:00.455+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:00.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:00.456+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:00.451+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:00.451+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:00.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:00.451+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:00.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:00.452+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:01.449 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:01.103+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:01.450 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:01.103+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:01.450 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:01.103+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:01.450 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:01.103+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:01.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:01.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:01.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:01.482+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:01.483+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:01.448+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:01.448+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:01.448+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:01.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:01.448+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:02.483 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:02.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:02.484 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:02.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:02.484 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:02.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:02.484 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:02.150+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:02.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:02.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:02.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:02.495+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:02.496+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:02.482+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:02.482+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:02.482+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:02.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:02.483+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:03.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:03.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:03.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:03.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:03.167+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:03.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:40:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:40:02.965+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29973 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:40:03.747 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:03.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:03.747 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:03.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:03.747 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:03.456+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:03.747 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:03.456+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:03.748 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:03.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:03.748 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:03.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:03.748 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:03.506+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:03.748 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:03.507+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:04.502 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:04.203+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:04.502 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:04.203+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:04.502 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:04.203+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:04.502 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:04.204+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:04.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:04.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:04.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:04.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:04.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:04.517+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:04.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:04.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:04.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:04.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:04.500+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:04.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:04.501+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:05.525 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:05.197+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:05.525 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:05.197+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:05.525 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:05.197+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:05.525 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:05.198+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:05.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:05.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:05.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:05.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:05.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:05.563+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:05.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:05.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:05.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:05.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:05.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:05.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:05.524+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:06.512 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:06.191+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:06.512 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:06.191+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:06.512 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:06.191+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:06.512 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:06.192+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:06.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:06.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:06.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:06.510+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:06.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:06.511+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:06.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:06.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:06.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:06.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:06.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:06.534+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:07.480 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:07.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:07.480 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:07.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:07.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:07.194+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:07.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:07.195+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:07.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:07.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:07.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:07.478+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:07.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:07.480+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:07.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:07.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:07.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:07.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:07.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:07.511+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:07.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:07.512+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:08.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:40:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:40:07.968+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29981 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:40:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:08.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:08.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:08.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:08.162+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:08.501+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:08.501+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:08.501+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:08.501+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:08.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:08.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:08.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:08.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:08.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:08.448+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:08.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:08.449+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:09.469 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:09.128+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:09.469 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:09.128+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:09.469 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:09.128+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:09.470 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:09.129+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:09.812 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:09.467+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:09.813 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:09.467+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:09.813 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:09.467+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:09.813 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:09.468+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:09.813 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:09.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:09.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:09.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:09.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:09.520+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:09.814 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:09.520+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:10.492 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:10.164+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:10.492 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:10.164+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:10.493 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:10.164+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:10.493 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:10.165+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:10.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:10.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:10.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:10.534+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:10.490+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:10.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:10.490+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:10.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:10.490+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:10.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:10.491+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:11.508 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:11.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:11.508 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:11.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:11.508 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:11.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:11.508 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:11.162+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:11.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:11.505+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:11.505+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:11.505+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:11.507+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:11.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:11.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:11.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:11.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:11.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:11.508+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:12.526 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:12.145+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:12.526 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:12.145+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:12.526 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:12.145+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:12.526 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:12.145+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:12.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:12.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:12.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:12.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:12.534+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:12.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:12.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:12.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:12.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:12.524+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:13.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:40:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:40:12.970+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29988 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:40:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:13.183+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:13.183+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:13.183+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:13.184+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:13.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:13.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:13.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:13.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:13.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:13.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:13.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:13.534+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:13.521+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:13.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:13.521+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:13.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:13.521+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:13.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:13.521+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:14.503 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:14.159+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:14.504 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:14.159+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:14.504 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:14.159+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:14.504 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:14.160+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:14.501+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:14.501+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:14.501+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:14.501+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:14.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:14.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:14.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:14.543+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:14.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:14.544+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:15.112+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:15.112+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:15.112+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:15.113+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:15.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:15.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:15.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:15.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:15.574+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:15.538+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:15.538+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:15.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:15.538+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:15.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:15.539+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:16.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:16.148+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:16.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:16.148+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:16.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:16.148+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:16.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:16.148+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:16.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:16.622+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:16.622+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:16.622+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:16.622+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:16.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:16.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:16.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:16.586+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:17.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:17.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:17.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:17.187+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:17.976 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:17.619+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:17.977 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:17.619+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:17.977 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:17.619+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:17.977 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:17.619+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:17.977 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:17.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:17.977 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:17.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:17.978 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:17.613+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:17.978 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:17.614+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:18.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:40:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:40:17.974+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 29996 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:40:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:18.138+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:18.138+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:18.138+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:18.139+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:18.597+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:18.597+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:18.597+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:18.598+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:18.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:18.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:18.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:18.593+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:19.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:19.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:19.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:19.094+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:19.095+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:20.028 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:19.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:20.029 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:19.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:20.029 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:19.585+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:20.029 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:19.586+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:20.029 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:19.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:20.029 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:19.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:20.030 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:19.588+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:20.030 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:19.589+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:20.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:20.052+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:20.052+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:20.052+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:20.053+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:20.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:20.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:20.578+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:20.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:20.578+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:20.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:20.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:20.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:20.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:20.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:20.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:20.594+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:21.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:21.008+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:21.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:21.008+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:21.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:21.008+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:21.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:21.009+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:21.997 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:21.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:21.997 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:21.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:21.997 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:21.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:21.998 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:21.621+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:21.998 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:21.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:21.998 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:21.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:21.998 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:21.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:21.998 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:21.617+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:22.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:21.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:21.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:21.995+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:21.996+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:22.968 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:22.639+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:22.969 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:22.639+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:22.969 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:22.639+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:22.969 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:22.639+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:22.969 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:22.653+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:22.969 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:22.653+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:22.970 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:22.653+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:22.970 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:22.653+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:23.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:40:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:40:22.978+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30003 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:40:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:22.967+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:22.967+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:22.967+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:22.968+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:23.954 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:23.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:23.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:23.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:23.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:23.614+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:23.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:23.615+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:23.955 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:23.628+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:23.955 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:23.628+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:23.956 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:23.628+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:23.956 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:23.629+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:24.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:23.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:23.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:23.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:23.953+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:24.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:24.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:24.955 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:24.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:24.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:24.582+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:24.956 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:24.583+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:24.956 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:24.619+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:24.956 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:24.619+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:24.956 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:24.619+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:24.956 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:24.620+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:24.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:24.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:24.953+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:24.955+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:25.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:25.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:25.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:25.572+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:25.617+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:25.617+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:25.617+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:25.618+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:26.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:25.906+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:25.906+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:25.906+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:26.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:25.906+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:26.936 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:26.618+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:26.936 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:26.618+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:26.937 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:26.618+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:26.937 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:26.618+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:26.937 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:26.641+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:26.937 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:26.641+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:26.937 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:26.641+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:26.937 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:26.641+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:27.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:26.934+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:26.934+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:26.934+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:26.935+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:27.893 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:27.612+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:27.893 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:27.612+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:27.893 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:27.612+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:27.893 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:27.613+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:27.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:27.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:27.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:27.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:27.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:27.631+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:27.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:27.632+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:28.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:40:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:40:27.981+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30011 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:40:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:27.891+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:27.891+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:27.891+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:27.892+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:28.885 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:28.649+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:28.885 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:28.649+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:28.886 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:28.649+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:28.886 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:28.650+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:28.886 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:28.611+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:28.886 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:28.611+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:28.886 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:28.611+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:28.886 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:28.611+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:29.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:28.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:28.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:28.883+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:28.884+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:29.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:29.563+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:29.563+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:29.563+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:29.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:29.564+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:29.660+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:29.660+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:29.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:29.660+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:29.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:29.661+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:30.210 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:29.851+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:30.211 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:29.851+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:30.211 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:29.851+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:30.211 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:29.852+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:30.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:30.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:30.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:30.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:30.669+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:30.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:30.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:30.545+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:30.545+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:31.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:30.899+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:30.899+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:30.899+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:30.899+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:31.946 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:31.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:31.946 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:31.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:31.947 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:31.580+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:31.947 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:31.581+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:31.947 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:31.658+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:31.947 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:31.658+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:31.947 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:31.658+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:31.947 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:31.659+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:32.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:31.944+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:32.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:31.944+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:31.944+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:31.945+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:32.543+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:32.543+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:32.543+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:32.544+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:32.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:32.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:32.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:32.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:32.621+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:32.911+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:32.911+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:32.911+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:32.911+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:33.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:40:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:40:32.984+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30018 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:40:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:33.565+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:33.565+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:33.565+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:33.566+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:33.641+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:33.641+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:33.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:33.641+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:33.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:33.642+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:34.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:33.879+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:33.879+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:33.879+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:33.880+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:34.893 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:34.627+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:34.894 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:34.627+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:34.894 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:34.627+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:34.894 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:34.628+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:34.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:34.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:34.894 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:34.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:34.895 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:34.586+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:34.895 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:34.586+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:35.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:34.892+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:35.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:34.892+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:35.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:34.892+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:35.297 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:34.892+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:35.863 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:35.618+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:35.864 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:35.618+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:35.864 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:35.618+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:35.864 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:35.618+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:35.864 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:35.598+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:35.864 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:35.598+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:35.864 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:35.598+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:35.865 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:35.598+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:36.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:35.861+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:35.861+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:35.861+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:35.862+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:36.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:36.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:36.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:36.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:36.652+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:36.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:36.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:36.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:36.569+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:37.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:36.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:36.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:36.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:36.866+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:37.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:37.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:37.627+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:37.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:37.628+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:37.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:37.566+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:37.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:37.566+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:37.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:37.566+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:37.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:37.566+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:37.854+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:37.854+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:37.854+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:37.855+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:38.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:40:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:40:37.987+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30026 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:40:38.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:38.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:38.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:38.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:38.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:38.634+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:38.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:38.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:38.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:38.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:38.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:38.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:38.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:38.568+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:39.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:38.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:38.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:38.902+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:39.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:38.903+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:39.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:39.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:39.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:39.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:39.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:39.684+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:39.526+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:39.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:39.526+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:39.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:39.526+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:39.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:39.527+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:40.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:39.946+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:39.946+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:39.946+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:39.947+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:40.659 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:40.483+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:40.659 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:40.483+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:40.660 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:40.483+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:40.660 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:40.484+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:40.993 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:40.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:40.993 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:40.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:40.994 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:40.657+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:40.994 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:40.658+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:41.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:40.991+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:40.991+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:40.991+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:40.992+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:41.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:41.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:41.651+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:41.651+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:41.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:41.461+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:41.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:41.461+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:41.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:41.461+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:41.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:41.462+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:42.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:42.031+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:42.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:42.031+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:42.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:42.031+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:42.032+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:42.673 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:42.414+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:42.673 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:42.414+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:42.673 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:42.414+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:42.673 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:42.415+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:42.992 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:42.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:42.992 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:42.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:42.992 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:42.671+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:42.992 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:42.672+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:42.999+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:42.999+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:42.999+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:43.000+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:43.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:40:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:40:42.990+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30033 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:40:43.677 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:43.422+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:43.677 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:43.422+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:43.677 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:43.422+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:43.678 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:43.423+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:43.971 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:43.676+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:43.971 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:43.676+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:43.971 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:43.676+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:43.971 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:43.676+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:44.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:43.969+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:43.969+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:43.969+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:44.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:43.970+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:44.660 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:44.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:44.660 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:44.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:44.661 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:44.392+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:44.661 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:44.392+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:44.980 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:44.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:44.981 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:44.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:44.981 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:44.658+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:44.981 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:44.659+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:44.979+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:44.979+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:44.979+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:44.980+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:45.631 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:45.437+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:45.631 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:45.437+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:45.631 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:45.437+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:45.632 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:45.437+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:45.977 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:45.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:45.977 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:45.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:45.977 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:45.629+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:45.977 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:45.630+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:46.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:45.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:45.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:45.975+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:45.976+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:46.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:46.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:46.596+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:46.597+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:46.487+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:46.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:46.487+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:46.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:46.487+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:46.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:46.488+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:47.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:46.951+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:46.951+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:46.951+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:46.952+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:47.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:47.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:47.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:47.647+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:47.648+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:47.498+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:47.498+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:47.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:47.498+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:47.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:47.498+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:48.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:40:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:40:47.993+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30041 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:40:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:47.943+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:47.943+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:47.943+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:47.944+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:48.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:48.486+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:48.486+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:48.486+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:48.487+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:48.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:48.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:48.661+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:48.662+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:48.925+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:48.925+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:48.925+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:48.926+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:49.675+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:49.675+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:49.675+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:49.676+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:49.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:49.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:49.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:49.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:49.527+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:49.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:49.527+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:50.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:49.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:49.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:49.958+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:49.959+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:50.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:50.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:50.653+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:50.654+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:50.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:50.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:50.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:50.557+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:50.833 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:50.557+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:51.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:51.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:51.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:51.005+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:51.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:51.640+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:51.640+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:51.640+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:51.641+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:51.556+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:51.556+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:51.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:51.556+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:51.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:51.557+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:52.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:52.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:52.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:52.040+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:52.041+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:52.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:52.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:52.656+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:52.657+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:52.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:52.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:52.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:52.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:52.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:52.563+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:53.076+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:53.076+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:53.076+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:53.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:53.077+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:53.332 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:40:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:40:52.995+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30048 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:40:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:53.551+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:53.551+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:53.551+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:53.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:53.552+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:53.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:53.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:53.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:53.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:53.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:53.616+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:53.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:53.616+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:54.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:54.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:54.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:54.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:54.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:54.050+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:54.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:54.534+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:54.534+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:54.534+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:54.535+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:54.572+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:54.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:54.572+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:54.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:54.572+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:54.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:54.573+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:55.518 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:55.097+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:55.519 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:55.097+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:55.519 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:55.097+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:55.519 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:55.097+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:55.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:55.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:55.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:55.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:55.603+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:55.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:55.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:55.516+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:55.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:55.517+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:56.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:56.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:56.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:56.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:56.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:56.065+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:56.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:56.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:56.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:56.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:56.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:56.569+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:56.834 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:56.569+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:56.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:56.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:56.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:56.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:56.834 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:56.533+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:56.835 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:56.534+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:57.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:57.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:57.064+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:57.064+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:57.498+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:57.498+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:57.498+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:57.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:57.499+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:57.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:57.539+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:57.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:57.539+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:57.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:57.539+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:57.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:57.540+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:58.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:40:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:40:57.999+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30056 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:40:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:58.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:58.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:58.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:58.016+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:58.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:58.458+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:58.458+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:58.458+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:58.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:58.459+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:58.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:58.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:58.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:58.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:58.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:58.590+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:40:59.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:59.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:59.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:59.016+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:40:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:40:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:40:59.017+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:59.507+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:59.507+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:59.507+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:40:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:40:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:40:59.508+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:40:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:59.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:59.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:59.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:59.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:59.603+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:40:59.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:40:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:40:59.604+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:00.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:00.014+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:00.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:00.014+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:00.014+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:00.014+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:00.803 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:00.492+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:00.803 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:00.492+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:00.803 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:00.492+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:00.803 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:00.493+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:00.803 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:00.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:00.803 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:00.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:00.804 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:00.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:00.804 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:00.642+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:01.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:01.022+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:01.022+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:01.022+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:01.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:01.023+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:01.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:01.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:01.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:01.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:01.522+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:01.639+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:01.639+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:01.639+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:01.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:01.639+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:02.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:02.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:02.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:02.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:02.062+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:02.509+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:02.509+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:02.509+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:02.509+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:02.655+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:02.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:02.655+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:02.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:02.655+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:02.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:02.656+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:03.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:41:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:41:03.003+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30063 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:41:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:03.104+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:03.104+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:03.104+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:03.104+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:03.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:03.519+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:03.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:03.519+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:03.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:03.519+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:03.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:03.520+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:03.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:03.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:03.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:03.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:03.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:03.704+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:04.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:04.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:04.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:04.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:04.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:04.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:04.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:04.148+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:04.750 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:04.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:04.751 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:04.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:04.751 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:04.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:04.751 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:04.480+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:05.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:04.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:04.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:04.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:04.749+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:05.472 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:05.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:05.472 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:05.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:05.472 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:05.122+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:05.472 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:05.122+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:05.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:05.470+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:05.723 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:05.470+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:05.723 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:05.470+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:05.723 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:05.471+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:05.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:05.720+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:05.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:05.720+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:05.723 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:05.720+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:05.724 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:05.721+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:06.433 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:06.123+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:06.433 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:06.123+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:06.433 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:06.123+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:06.434 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:06.123+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:06.765 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:06.431+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:06.766 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:06.431+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:06.766 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:06.431+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:06.766 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:06.432+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:07.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:06.764+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:06.764+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:06.764+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:06.764+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:07.466 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:07.119+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:07.466 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:07.119+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:07.466 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:07.119+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:07.466 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:07.119+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:07.739 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:07.464+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:07.739 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:07.464+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:07.739 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:07.464+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:07.739 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:07.465+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:08.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:07.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:08.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:07.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:08.007 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:07.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:08.008 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:07.738+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:08.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:41:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:41:08.005+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30071 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:41:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:08.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:08.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:08.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:08.158+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:08.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:08.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:08.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:08.703+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:08.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:08.704+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:08.475+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:08.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:08.475+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:08.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:08.475+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:08.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:08.475+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:09.472 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:09.121+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:09.473 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:09.121+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:09.473 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:09.121+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:09.473 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:09.122+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:09.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:09.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:09.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:09.714+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:09.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:09.715+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:09.471+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:09.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:09.471+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:09.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:09.471+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:09.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:09.472+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:10.483 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:10.113+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:10.483 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:10.113+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:10.483 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:10.113+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:10.484 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:10.114+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:10.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:10.481+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:10.481+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:10.481+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:10.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:10.483+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:10.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:10.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:10.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:10.681+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:10.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:10.682+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:11.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:11.078+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:11.078+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:11.078+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:11.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:11.079+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:11.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:11.455+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:11.455+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:11.455+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:11.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:11.455+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:11.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:11.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:11.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:11.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:11.644+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:11.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:11.645+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:12.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:12.039+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:12.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:12.039+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:12.039+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:12.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:12.040+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:12.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:12.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:12.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:12.602+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:12.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:12.603+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:12.486+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:12.486+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:12.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:12.486+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:12.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:12.487+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:13.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:41:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:41:13.009+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30079 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:41:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:13.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:13.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:13.015+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:13.016+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:13.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:13.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:13.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:13.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:13.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:13.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:13.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:13.530+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:13.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:13.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:13.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:13.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:13.622+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:13.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:13.622+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:14.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:14.065+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:14.065+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:14.065+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:14.066+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:14.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:14.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:14.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:14.545+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:14.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:14.545+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:14.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:14.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:14.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:14.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:14.610+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:14.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:14.611+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:15.030+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:15.030+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:15.030+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:15.031+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:15.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:15.572+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:15.572+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:15.572+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:15.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:15.573+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:15.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:15.553+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:15.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:15.553+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:15.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:15.553+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:15.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:15.554+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:16.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:15.983+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:15.983+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:15.983+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:16.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:15.983+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:16.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:16.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:16.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:16.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:16.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:16.569+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:16.536+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:16.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:16.536+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:16.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:16.536+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:16.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:16.537+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:17.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:17.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:17.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:17.004+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:17.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:17.005+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:17.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:17.606+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:17.606+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:17.606+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:17.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:17.606+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:17.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:17.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:17.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:17.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:17.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:17.571+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:17.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:17.571+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:18.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:41:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:41:18.012+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30087 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:41:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:17.971+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:17.971+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:17.971+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:18.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:17.972+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:18.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:18.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:18.568+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:18.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:18.569+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:18.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:18.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:18.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:18.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:18.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:18.577+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:18.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:18.578+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:18.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:18.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:18.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:19.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:18.994+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:19.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:19.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:19.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:19.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:19.562+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:19.537+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:19.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:19.537+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:19.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:19.537+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:19.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:19.538+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:20.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:20.041+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:20.041+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:20.041+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:20.041+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:20.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:20.607+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:20.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:20.607+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:20.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:20.607+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:20.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:20.608+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:20.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:20.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:20.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:20.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:20.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:20.515+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:20.833 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:20.516+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:21.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:21.030+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:21.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:21.030+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:21.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:21.030+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:21.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:21.031+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:21.564+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:21.564+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:21.564+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:21.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:21.564+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:21.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:21.540+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:21.540+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:21.540+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:21.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:21.541+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:22.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:22.046+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:22.046+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:22.046+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:22.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:22.047+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:22.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:22.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:22.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:22.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:22.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:22.557+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:22.523+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:22.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:22.523+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:22.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:22.523+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:22.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:22.524+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:23.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:41:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:41:23.015+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30094 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:41:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:23.053+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:23.053+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:23.053+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:23.053+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:23.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:23.524+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:23.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:23.524+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:23.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:23.524+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:23.753 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:23.525+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:23.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:23.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:23.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:23.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:23.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:23.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:23.754 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:23.480+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:24.320 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:24.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:24.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:24.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:24.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:24.020+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:24.321 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:24.020+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:24.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:24.512+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:24.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:24.512+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:24.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:24.512+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:24.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:24.513+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:24.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:24.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:24.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:24.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:24.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:24.502+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:24.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:24.502+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:25.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:24.979+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:24.979+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:24.979+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:25.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:24.980+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:25.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:25.477+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:25.477+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:25.477+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:25.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:25.477+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:25.556+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:25.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:25.556+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:25.556+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:25.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:25.556+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:26.282 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:25.952+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:26.282 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:25.952+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:26.283 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:25.952+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:26.283 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:25.952+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:26.565 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:26.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:26.565 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:26.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:26.566 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:26.480+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:26.566 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:26.481+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:26.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:26.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:26.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:26.562+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:26.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:26.563+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:27.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:26.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:26.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:26.940+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:27.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:26.941+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:27.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:27.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:27.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:27.507+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:27.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:27.507+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:27.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:27.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:27.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:27.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:27.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:27.573+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:28.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:41:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:41:28.017+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30102 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:41:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:27.932+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:27.932+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:27.932+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:27.933+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:28.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:28.563+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:28.563+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:28.563+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:28.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:28.564+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:28.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:28.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:28.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:28.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:28.523+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:28.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:28.524+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:29.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:28.942+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:28.942+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:28.942+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:29.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:28.943+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:29.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:29.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:29.587+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:29.587+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:29.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:29.560+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:29.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:29.560+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:29.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:29.560+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:29.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:29.560+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:30.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:29.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:29.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:29.966+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:29.967+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:30.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:30.560+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:30.560+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:30.560+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:30.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:30.561+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:30.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:30.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:30.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:30.554+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:30.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:30.555+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:31.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:30.967+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:30.967+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:30.967+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:30.968+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:31.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:31.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:31.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:31.516+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:31.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:31.517+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:31.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:31.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:31.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:31.521+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:31.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:31.522+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:31.974+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:31.974+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:31.974+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:32.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:31.975+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:32.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:32.481+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:32.481+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:32.481+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:32.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:32.482+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:32.565+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:32.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:32.565+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:32.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:32.565+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:32.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:32.566+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:33.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:32.943+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:32.943+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:32.943+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:33.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:32.944+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:33.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:41:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:41:33.021+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30109 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:41:33.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:33.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:33.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:33.512+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:33.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:33.513+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:33.615+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:33.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:33.615+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:33.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:33.615+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:33.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:33.615+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:34.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:33.896+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:33.896+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:33.896+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:34.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:33.896+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:34.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:34.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:34.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:34.509+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:34.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:34.509+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:34.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:34.660+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:34.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:34.660+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:34.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:34.660+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:34.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:34.661+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:35.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:34.849+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:34.849+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:34.849+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:34.850+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:35.656+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:35.656+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:35.656+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:35.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:35.656+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:35.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:35.558+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:35.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:35.558+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:35.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:35.558+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:35.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:35.559+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:36.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:35.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:35.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:35.875+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:36.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:35.876+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:36.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:36.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:36.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:36.556+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:36.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:36.556+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:36.635+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:36.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:36.635+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:36.635+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:36.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:36.636+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:37.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:36.853+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:36.853+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:36.853+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:37.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:36.854+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:38.026 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:37.819+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:38.027 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:37.819+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:38.027 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:37.819+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:38.027 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:37.820+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:38.027 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:37.588+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:38.027 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:37.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:38.028 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:37.589+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:38.028 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:37.589+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:38.028 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:37.615+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:38.028 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:37.615+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:38.028 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:37.615+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:38.029 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:37.615+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:38.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:41:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:41:38.024+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30117 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:41:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:38.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:38.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:38.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:38.863+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:39.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:38.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:39.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:38.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:39.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:38.620+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:39.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:38.621+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:39.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:38.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:39.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:38.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:39.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:38.634+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:39.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:38.635+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:39.911 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:39.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:39.911 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:39.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:39.911 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:39.628+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:39.911 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:39.629+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:39.912 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:39.603+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:39.912 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:39.603+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:39.912 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:39.603+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:39.912 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:39.604+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:40.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:39.909+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:39.909+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:39.909+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:40.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:39.910+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:40.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:40.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:40.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:40.641+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:40.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:40.642+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:40.577+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:40.577+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:40.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:40.577+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:40.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:40.577+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:40.917+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:40.917+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:40.917+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:41.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:40.918+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:41.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:41.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:41.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:41.670+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:41.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:41.671+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:41.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:41.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:41.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:41.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:41.573+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:41.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:41.574+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:41.907+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:41.907+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:41.907+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:42.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:41.908+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:42.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:42.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:42.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:42.693+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:42.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:42.694+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:42.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:42.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:42.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:42.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:42.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:42.530+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:43.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:41:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:41:43.027+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30124 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:41:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:42.900+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:42.900+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:42.900+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:42.901+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:43.755 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:43.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:43.755 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:43.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:43.755 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:43.696+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:43.755 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:43.697+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:43.756 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:43.496+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:43.756 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:43.496+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:43.756 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:43.496+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:43.756 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:43.496+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:44.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:43.890+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:43.890+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:43.890+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:43.890+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:44.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:44.478+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:44.478+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:44.478+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:44.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:44.479+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:44.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:44.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:44.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:44.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:44.650+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:44.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:44.651+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:45.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:44.918+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:44.918+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:44.918+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:45.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:44.919+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:45.670 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:45.513+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:45.670 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:45.513+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:45.671 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:45.513+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:45.671 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:45.514+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:45.952 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:45.668+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:45.953 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:45.668+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:45.953 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:45.668+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:45.953 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:45.670+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:46.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:45.951+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:45.951+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:45.951+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:46.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:45.952+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:46.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:46.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:46.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:46.649+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:46.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:46.649+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:46.523+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:46.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:46.523+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:46.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:46.523+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:46.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:46.524+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:46.971+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:46.971+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:46.971+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:47.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:46.972+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:47.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:47.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:47.607+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:47.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:47.608+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:47.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:47.569+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:47.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:47.569+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:47.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:47.569+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:47.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:47.569+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:48.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:41:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:41:48.030+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30132 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:41:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:48.001+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:48.001+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:48.001+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:48.001+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:48.574+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:48.574+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:48.574+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:48.574+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:48.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:48.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:48.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:48.633+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:48.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:48.634+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:49.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:49.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:49.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:49.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:49.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:49.036+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:49.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:49.563+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:49.563+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:49.563+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:49.564+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:49.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:49.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:49.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:49.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:49.612+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:49.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:49.613+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:50.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:50.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:50.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:50.049+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:50.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:50.050+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:50.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:50.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:50.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:50.593+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:50.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:50.594+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:50.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:50.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:50.529+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:50.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:50.529+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:51.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:51.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:51.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:51.029+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:51.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:51.029+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:51.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:51.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:51.525+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:51.526+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:51.611+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:51.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:51.611+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:51.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:51.611+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:51.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:51.611+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:52.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:51.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:51.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:51.993+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:51.994+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:52.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:52.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:52.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:52.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:52.503+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:52.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:52.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:52.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:52.648+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:52.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:52.649+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:52.994+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:52.994+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:53.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:52.994+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:53.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:52.995+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:53.332 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:41:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:41:53.033+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30139 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:41:53.756 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:53.484+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:53.756 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:53.484+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:53.756 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:53.484+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:53.756 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:53.484+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:53.757 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:53.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:53.757 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:53.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:53.757 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:53.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:53.757 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:53.678+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:54.001+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:54.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:54.002+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:54.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:54.002+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:54.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:54.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:54.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:54.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:54.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:54.708+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:54.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:54.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:54.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:54.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:54.479+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:54.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:54.480+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:55.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:55.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:55.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:55.035+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:55.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:55.036+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:55.741 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:55.452+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:55.741 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:55.452+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:55.742 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:55.452+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:55.742 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:55.453+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:56.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:55.740+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:55.740+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:55.740+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:55.741+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:56.499 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:56.082+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:56.500 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:56.082+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:56.500 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:56.082+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:56.500 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:56.082+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:56.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:56.498+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:56.498+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:56.498+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:56.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:56.498+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:56.726+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:56.726+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:56.726+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:56.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:56.727+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:57.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:57.092+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:57.092+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:57.092+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:57.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:57.093+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:57.776 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:57.503+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:57.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:57.503+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:57.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:57.503+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:57.777 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:57.504+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:58.040 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:57.774+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:58.041 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:57.774+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:58.041 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:57.774+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:58.041 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:57.775+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:58.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:41:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:41:58.036+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30147 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:41:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:58.131+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:58.131+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:58.131+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:58.132+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:58.735 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:58.483+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:58.735 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:58.483+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:58.735 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:58.483+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:58.736 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:58.484+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:58.733+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:58.733+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:58.733+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:59.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:58.734+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:41:59.504 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:59.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:59.504 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:59.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:59.505 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:59.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:41:59.505 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:41:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:41:59.083+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:59.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:59.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:59.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:59.502+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:41:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:41:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:41:59.503+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:41:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:59.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:59.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:59.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:59.734+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:41:59.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:41:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:41:59.735+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:00.468 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:00.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:00.469 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:00.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:00.469 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:00.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:00.469 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:00.090+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:00.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:00.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:00.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:00.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:00.711+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:00.467+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:00.467+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:00.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:00.467+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:00.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:00.467+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:01.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:01.051+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:01.051+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:01.051+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:01.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:01.051+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:01.711 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:01.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:01.711 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:01.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:01.711 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:01.424+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:01.711 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:01.425+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:01.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:01.709+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:01.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:01.709+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:01.991 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:01.709+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:01.992 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:01.710+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:02.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:02.073+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:02.073+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:02.073+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:02.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:02.074+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:02.703 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:02.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:02.704 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:02.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:02.704 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:02.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:02.704 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:02.394+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:03.041 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:02.702+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:03.041 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:02.702+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:03.042 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:02.702+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:03.042 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:02.703+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:03.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:42:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:42:03.038+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30154 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:42:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:03.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:03.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:03.072+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:03.073+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:03.712 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:03.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:03.712 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:03.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:03.713 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:03.411+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:03.713 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:03.412+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:04.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:03.709+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:03.709+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:03.709+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:03.710+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:04.405 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:04.113+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:04.406 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:04.113+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:04.406 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:04.113+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:04.406 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:04.114+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:04.700 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:04.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:04.700 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:04.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:04.701 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:04.402+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:04.701 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:04.403+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:05.074 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:04.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:05.075 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:04.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:05.075 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:04.697+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:05.075 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:04.698+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:05.071+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:05.071+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:05.071+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:05.072+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:05.687 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:05.397+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:05.687 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:05.397+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:05.687 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:05.397+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:05.687 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:05.398+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:06.076 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:05.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:06.077 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:05.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:06.077 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:05.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:06.077 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:05.685+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:06.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:06.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:06.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:06.074+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:06.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:06.074+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:06.680 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:06.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:06.680 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:06.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:06.681 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:06.394+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:06.681 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:06.394+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:07.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:06.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:06.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:06.677+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:06.678+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:07.436 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:07.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:07.437 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:07.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:07.437 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:07.124+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:07.437 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:07.125+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:07.722 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:07.433+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:07.723 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:07.433+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:07.723 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:07.433+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:07.723 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:07.434+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:08.044 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:07.720+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:08.044 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:07.720+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:08.044 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:07.720+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:08.044 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:07.720+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:08.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:42:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:42:08.041+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30162 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:42:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:08.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:08.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:08.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:08.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:08.163+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:08.731 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:08.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:08.732 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:08.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:08.732 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:08.389+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:08.732 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:08.389+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:09.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:08.728+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:08.728+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:08.728+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:08.730+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:09.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:09.198+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:09.198+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:09.198+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:09.199+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:09.341+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:09.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:09.341+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:09.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:09.341+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:09.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:09.342+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:10.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:09.745+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:09.745+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:09.745+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:10.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:09.746+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:10.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:10.224+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:10.224+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:10.224+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:10.225+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:10.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:10.372+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:10.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:10.372+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:10.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:10.372+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:10.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:10.373+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:10.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:10.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:10.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:11.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:10.793+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:11.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:11.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:11.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:11.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:11.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:11.361+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:11.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:11.362+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:11.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:11.232+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:11.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:11.232+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:11.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:11.232+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:11.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:11.233+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:12.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:11.752+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:11.752+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:11.752+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:12.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:11.753+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:12.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:12.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:12.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:12.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:12.229+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:12.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:12.312+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:12.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:12.312+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:12.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:12.312+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:12.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:12.313+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:13.047 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:12.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:13.047 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:12.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:13.047 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:12.710+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:13.048 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:12.711+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:13.326 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:42:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:42:13.045+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30169 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:42:13.326 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:13.192+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:13.327 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:13.192+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:13.327 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:13.192+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:13.327 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:13.193+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:13.324+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:13.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:13.324+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:13.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:13.324+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:13.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:13.325+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:14.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:13.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:14.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:13.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:14.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:13.684+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:14.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:13.684+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:14.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:14.171+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:14.171+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:14.171+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:14.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:14.172+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:14.340+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:14.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:14.340+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:14.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:14.340+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:14.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:14.341+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:15.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:14.728+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:15.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:14.728+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:15.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:14.728+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:15.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:14.728+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:15.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:15.206+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:15.206+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:15.206+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:15.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:15.207+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:15.313+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:15.313+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:15.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:15.313+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:15.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:15.313+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:16.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:15.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:16.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:15.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:16.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:15.699+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:16.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:15.700+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:16.544 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:16.278+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:16.545 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:16.278+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:16.545 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:16.278+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:16.545 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:16.279+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:16.545 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:16.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:16.545 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:16.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:16.546 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:16.162+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:16.546 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:16.163+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:16.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:16.747+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:16.747+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:16.747+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:16.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:16.748+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:17.145+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:17.145+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:17.145+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:17.146+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:17.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:17.287+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:17.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:17.287+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:17.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:17.287+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:17.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:17.287+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:18.050 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:17.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:18.051 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:17.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:18.051 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:17.748+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:18.051 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:17.749+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:18.305 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:18.140+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:18.306 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:18.140+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:18.306 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:18.140+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:18.306 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:18.141+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:18.306 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:42:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:42:18.048+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30177 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:42:18.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:18.304+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:18.304+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:18.304+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:18.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:18.305+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:19.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:18.778+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:18.778+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:18.778+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:18.779+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:19.306+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:19.306+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:19.306+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:19.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:19.307+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:19.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:19.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:19.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:19.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:19.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:19.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:19.177+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:20.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:19.756+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:19.756+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:19.756+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:20.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:19.757+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:20.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:20.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:20.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:20.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:20.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:20.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:20.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:20.151+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:20.270+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:20.270+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:20.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:20.270+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:20.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:20.271+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:21.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:20.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:20.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:20.773+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:21.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:20.774+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:21.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:21.111+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:21.111+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:21.111+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:21.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:21.111+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:21.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:21.276+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:21.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:21.276+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:21.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:21.276+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:21.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:21.277+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:21.733+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:21.733+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:21.733+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:22.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:21.733+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:22.380 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:22.105+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:22.380 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:22.105+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:22.380 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:22.105+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:22.380 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:22.106+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:22.381 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:22.265+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:22.381 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:22.265+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:22.381 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:22.265+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:22.381 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:22.266+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:23.053 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:22.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:23.053 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:22.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:23.053 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:22.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:23.053 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:22.709+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:23.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:42:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:42:23.051+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30184 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:42:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:23.134+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:23.134+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:23.134+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:23.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:23.134+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:23.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:23.298+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:23.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:23.298+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:23.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:23.298+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:23.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:23.298+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:23.753+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:23.753+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:24.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:23.753+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:24.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:23.753+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:24.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:24.111+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:24.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:24.111+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:24.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:24.111+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:24.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:24.112+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:24.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:24.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:24.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:24.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:24.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:24.283+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:24.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:24.284+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:25.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:24.783+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:25.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:24.783+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:25.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:24.783+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:25.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:24.783+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:25.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:25.264+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:25.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:25.264+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:25.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:25.264+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:25.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:25.265+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:25.111+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:25.111+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:25.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:25.111+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:25.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:25.111+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:26.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:25.820+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:25.820+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:25.820+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:26.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:25.821+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:26.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:26.136+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:26.136+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:26.136+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:26.137+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:26.244+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:26.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:26.244+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:26.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:26.244+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:26.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:26.245+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:27.156 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:26.842+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:27.156 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:26.842+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:27.157 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:26.842+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:27.157 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:26.843+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:27.473 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:27.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:27.473 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:27.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:27.473 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:27.155+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:27.474 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:27.155+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:27.474 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:27.201+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:27.474 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:27.201+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:27.474 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:27.201+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:27.474 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:27.202+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:28.056 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:27.796+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:28.056 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:27.796+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:28.056 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:27.796+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:28.056 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:27.797+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:28.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:42:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:42:28.054+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30192 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:42:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:28.121+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:28.121+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:28.121+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:28.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:28.122+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:28.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:28.188+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:28.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:28.188+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:28.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:28.188+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:28.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:28.189+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:29.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:28.765+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:28.765+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:28.765+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:28.765+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:29.108+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:29.108+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:29.108+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:29.109+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:29.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:29.163+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:29.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:29.163+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:29.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:29.163+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:29.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:29.164+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:30.063 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:29.808+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:30.063 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:29.808+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:30.063 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:29.808+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:30.063 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:29.809+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:30.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:30.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:30.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:30.061+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:30.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:30.061+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:30.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:30.146+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:30.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:30.146+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:30.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:30.146+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:30.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:30.147+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:31.061 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:30.807+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:31.061 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:30.807+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:31.061 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:30.807+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:31.062 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:30.807+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:31.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:31.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:31.059+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:31.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:31.060+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:31.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:31.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:31.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:31.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:31.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:31.169+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:31.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:31.170+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:31.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:31.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:31.814+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:32.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:31.815+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:32.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:32.086+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:32.086+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:32.086+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:32.086+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:32.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:32.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:32.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:32.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:32.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:32.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:32.204+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:33.134 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:42:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:42:33.057+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30199 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:42:33.134 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:32.855+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:33.134 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:32.855+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:33.135 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:32.855+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:33.135 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:32.855+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:33.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:33.132+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:33.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:33.132+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:33.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:33.132+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:33.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:33.133+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:33.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:33.182+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:33.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:33.182+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:33.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:33.182+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:33.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:33.183+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:34.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:33.809+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:33.809+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:33.809+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:34.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:33.810+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:34.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:34.205+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:34.205+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:34.205+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:34.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:34.205+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:34.158+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:34.158+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:34.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:34.158+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:34.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:34.159+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:34.976 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:34.804+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:34.977 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:34.804+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:34.977 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:34.804+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:34.977 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:34.805+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:35.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:35.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:35.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:35.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:35.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:35.168+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:35.212+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:35.212+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:35.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:35.212+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:35.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:35.213+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:36.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:35.786+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:35.786+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:35.786+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:35.787+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:36.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:36.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:36.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:36.152+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:36.153+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:36.166+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:36.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:36.166+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:36.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:36.166+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:36.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:36.167+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:37.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:36.739+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:37.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:36.739+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:37.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:36.739+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:37.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:36.740+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:37.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:37.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:37.147+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:37.148+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:37.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:37.200+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:37.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:37.200+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:37.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:37.200+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:37.583 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:37.200+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:38.062 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:37.783+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:38.063 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:37.783+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:38.063 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:37.783+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:38.063 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:37.784+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:38.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:42:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:42:38.061+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30207 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:42:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:38.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:38.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:38.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:38.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:38.128+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:38.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:38.239+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:38.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:38.239+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:38.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:38.239+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:38.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:38.240+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:39.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:38.746+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:38.746+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:38.746+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:38.747+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:39.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:39.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:39.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:39.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:39.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:39.142+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:39.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:39.143+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:39.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:39.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:39.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:39.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:39.204+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:39.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:39.204+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:40.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:39.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:39.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:39.793+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:40.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:39.794+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:40.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:40.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:40.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:40.162+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:40.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:40.232+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:40.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:40.232+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:40.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:40.232+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:40.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:40.233+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:41.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:40.794+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:40.794+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:40.794+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:41.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:40.795+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:41.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:41.125+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:41.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:41.125+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:41.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:41.125+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:41.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:41.126+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:41.227+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:41.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:41.227+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:41.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:41.227+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:41.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:41.228+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:42.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:41.766+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:41.766+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:41.766+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:42.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:41.767+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:42.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:42.125+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:42.125+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:42.125+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:42.126+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:42.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:42.246+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:42.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:42.246+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:42.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:42.246+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:42.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:42.246+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:43.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:42.799+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:43.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:42.799+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:43.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:42.799+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:43.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:42.799+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:43.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:42:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:42:43.063+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30214 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:42:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:43.150+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:43.150+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:43.150+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:43.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:43.151+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:43.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:43.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:43.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:43.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:43.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:43.223+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:43.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:43.223+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:44.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:43.842+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:43.842+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:43.842+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:44.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:43.842+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:44.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:44.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:44.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:44.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:44.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:44.186+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:44.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:44.187+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:44.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:44.198+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:44.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:44.198+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:44.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:44.198+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:44.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:44.198+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:45.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:44.815+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:45.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:44.815+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:45.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:44.815+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:45.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:44.816+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:45.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:45.153+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:45.153+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:45.153+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:45.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:45.154+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:45.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:45.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:45.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:45.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:45.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:45.203+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:45.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:45.204+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:46.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:45.807+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:45.807+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:45.807+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:46.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:45.807+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:46.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:46.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:46.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:46.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:46.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:46.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:46.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:46.121+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:46.219+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:46.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:46.219+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:46.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:46.219+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:46.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:46.219+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:47.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:46.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:46.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:46.792+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:47.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:46.792+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:47.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:47.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:47.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:47.161+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:47.162+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:47.267+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:47.267+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:47.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:47.267+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:47.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:47.268+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:48.068 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:47.797+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:48.069 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:47.797+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:48.069 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:47.797+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:48.069 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:47.798+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:48.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:42:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:42:48.067+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30222 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:42:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:48.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:48.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:48.176+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:48.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:48.176+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:48.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:48.246+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:48.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:48.246+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:48.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:48.246+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:48.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:48.247+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:49.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:48.769+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:48.769+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:48.769+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:49.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:48.770+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:49.205+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:49.205+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:49.205+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:49.205+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:49.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:49.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:49.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:49.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:49.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:49.197+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:49.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:49.198+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:50.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:49.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:49.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:49.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:50.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:49.755+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:50.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:50.237+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:50.237+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:50.237+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:50.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:50.238+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:50.170+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:50.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:50.170+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:50.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:50.170+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:50.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:50.171+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:51.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:50.784+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:51.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:50.784+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:51.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:50.784+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:51.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:50.784+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:51.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:51.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:51.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:51.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:51.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:51.229+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:51.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:51.229+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:51.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:51.123+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:51.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:51.123+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:51.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:51.123+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:51.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:51.124+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:52.079 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:51.798+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:52.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:51.798+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:52.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:51.798+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:52.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:51.799+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:52.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:52.184+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:52.184+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:52.184+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:52.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:52.184+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:52.078+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:52.078+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:52.078+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:52.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:52.079+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:53.072 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:52.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:53.072 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:52.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:53.073 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:52.759+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:53.073 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:52.759+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:53.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:53.208+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:53.208+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:53.208+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:53.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:53.208+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:53.127+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:53.127+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:53.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:53.127+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:53.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:53.128+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:53.332 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:42:53.070+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30229 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:42:54.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:53.755+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:54.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:53.755+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:54.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:53.755+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:54.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:53.756+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:54.169+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:54.169+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:54.169+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:54.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:54.170+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:54.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:54.124+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:54.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:54.124+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:54.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:54.124+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:54.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:54.124+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:55.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:54.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:55.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:54.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:55.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:54.737+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:55.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:54.738+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:55.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:55.148+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:55.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:55.148+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:55.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:55.148+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:55.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:55.149+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:55.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:55.172+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:55.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:55.172+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:55.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:55.172+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:55.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:55.173+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:56.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:55.726+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:55.726+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:55.726+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:56.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:55.727+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:56.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:56.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:56.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:56.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:56.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:56.168+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:56.481 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:56.169+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:56.481 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:56.129+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:56.482 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:56.129+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:56.482 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:56.129+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:56.482 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:56.130+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:56.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:56.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:56.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:56.682+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:56.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:56.683+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:57.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:57.203+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:57.203+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:57.203+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:57.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:57.204+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:57.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:57.083+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:57.581 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:57.083+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:57.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:57.083+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:57.582 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:57.084+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:58.065 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:57.687+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:58.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:57.687+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:58.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:57.687+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:58.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:57.688+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:58.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:58.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:58.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:58.163+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:58.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:58.164+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:58.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:58.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:58.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:58.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:58.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:58.064+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:58.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:58.065+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:58.333 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:42:58.073+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30237 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:42:59.043 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:58.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:59.043 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:58.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:59.044 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:58.708+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:42:59.044 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:58.709+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:42:59.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:59.121+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:59.121+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:59.121+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:42:59.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:42:59.122+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:42:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:59.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:59.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:59.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:59.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:59.042+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:42:59.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:42:59.043+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:00.014 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:59.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:00.014 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:59.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:00.014 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:59.735+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:00.014 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:42:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:42:59.735+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:00.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:00.117+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:00.117+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:00.117+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:00.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:00.118+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:00.012+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:00.012+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:00.012+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:00.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:00.013+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:00.976+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:00.976+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:00.976+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:01.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:00.977+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:01.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:00.769+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:01.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:00.769+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:01.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:00.769+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:01.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:00.770+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:01.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:01.153+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:01.153+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:01.153+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:01.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:01.154+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:01.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:01.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:01.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:02.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:01.979+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:02.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:01.819+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:02.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:01.819+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:02.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:01.819+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:02.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:01.820+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:02.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:02.178+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:02.178+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:02.178+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:02.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:02.179+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:03.078 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:02.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:03.078 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:02.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:03.078 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:02.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:03.078 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:02.822+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:03.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:02.974+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:03.079 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:02.974+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:03.079 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:02.974+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:03.079 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:02.974+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:03.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:43:03.076+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30244 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:43:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:03.225+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:03.225+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:03.225+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:03.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:03.225+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:03.962+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:03.962+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:03.962+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:04.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:03.962+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:04.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:03.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:04.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:03.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:04.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:03.823+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:04.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:03.824+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:04.239+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:04.239+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:04.239+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:04.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:04.240+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:04.986 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:04.803+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:04.987 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:04.803+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:04.987 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:04.803+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:04.987 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:04.804+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:05.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:05.213+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:05.213+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:05.213+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:05.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:05.214+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:04.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:05.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:04.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:05.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:04.985+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:05.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:04.985+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:06.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:05.779+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:05.779+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:05.779+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:06.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:05.779+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:06.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:05.938+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:06.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:05.938+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:06.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:05.938+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:06.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:05.938+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:06.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:06.201+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:06.201+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:06.201+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:06.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:06.202+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:06.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:06.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:06.754+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:07.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:06.754+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:07.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:06.909+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:07.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:06.909+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:07.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:06.909+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:07.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:06.910+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:07.192+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:07.192+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:07.192+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:07.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:07.192+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:07.778+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:07.778+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:07.778+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:08.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:07.779+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:08.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:07.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:08.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:07.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:08.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:07.881+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:08.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:07.882+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:08.196+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:08.196+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:08.196+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:08.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:08.196+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:08.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:43:08.079+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30252 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:43:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:08.790+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:08.790+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:08.790+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:09.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:08.791+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:09.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:08.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:09.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:08.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:09.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:08.878+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:09.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:08.879+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:09.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:09.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:09.157+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:09.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:09.157+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:10.140 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:09.834+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:10.141 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:09.834+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:10.141 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:09.834+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:10.141 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:09.834+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:10.141 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:10.139+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:10.142 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:09.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:10.142 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:09.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:10.142 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:09.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:10.142 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:09.870+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:10.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:10.139+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:10.139+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:10.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:10.139+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:11.137 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:10.829+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:11.138 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:10.829+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:11.138 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:10.829+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:11.138 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:10.830+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:11.138 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:10.868+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:11.138 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:10.868+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:11.138 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:10.868+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:11.138 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:10.868+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:11.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:11.135+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:11.135+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:11.135+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:11.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:11.136+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:12.090 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:11.833+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:12.090 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:11.833+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:12.090 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:11.833+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:12.091 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:11.833+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:12.091 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:11.868+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:12.091 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:11.868+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:12.091 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:11.868+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:12.091 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:11.868+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:12.573 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-09-10T04:43:12.579 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T04:43:12.579 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-09-10T04:43:12.580 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-09-10T04:43:12.594 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-10T04:43:12.606 INFO:tasks.cephadm:Teardown begin 2024-09-10T04:43:12.606 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephadm.py", line 2374, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-09-10T04:43:12.611 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-10T04:43:12.614 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:12.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:12.615 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:12.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:12.615 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:12.088+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:12.615 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:12.089+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:12.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:43:12.644 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:43:13.058 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:12.791+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:13.059 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:12.791+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:13.059 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:12.791+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:13.059 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:12.792+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:13.059 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:12.860+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:13.059 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:12.860+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:13.060 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:12.860+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:13.060 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:12.861+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:13.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:43:13.082+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30259 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:43:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:13.057+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:13.057+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:13.057+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:13.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:13.058+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:14.069 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:13.868+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:14.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:13.868+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:14.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:13.868+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:14.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:13.869+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:14.070 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:13.830+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:14.070 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:13.830+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:14.070 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:13.830+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:14.071 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:13.831+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:14.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:14.067+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:14.067+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:14.067+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:14.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:14.068+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:14.913 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:14.828+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:14.913 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:14.828+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:14.913 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:14.828+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:14.914 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:14.829+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:14.914 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:14.806+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:14.914 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:14.806+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:14.914 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:14.806+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:14.914 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:14.806+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:15.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:15.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:15.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:15.062+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:15.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:15.063+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:15.709 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:43:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:15.848+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:15.848+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:15.848+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:15.849+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:16.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:15.801+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:15.801+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:15.801+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:16.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:15.802+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:16.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:16.087+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:16.087+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:16.087+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:16.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:16.087+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:17.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:16.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:16.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:16.844+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:16.845+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:17.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:16.811+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:17.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:16.811+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:17.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:16.811+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:17.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:16.812+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:17.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:17.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:17.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:17.106+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:17.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:17.107+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:18.086 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:17.832+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:18.086 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:17.832+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:18.087 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:17.832+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:18.087 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:17.832+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:18.087 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:17.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:18.087 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:17.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:18.087 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:17.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:18.087 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:17.833+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:18.542 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:43:18.084+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30267 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:43:18.542 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:18.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:18.542 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:18.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:18.543 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:18.083+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:18.543 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:18.084+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:19.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:18.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:18.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:18.882+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:19.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:18.883+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:18.819+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:19.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:18.819+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:18.819+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:19.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:18.820+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:19.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:19.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:19.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:19.089+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:19.090+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:20.068+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:20.068+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:20.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:20.068+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:20.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:20.068+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:20.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:19.891+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:20.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:19.891+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:20.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:19.891+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:20.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:19.892+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:20.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:19.860+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:20.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:19.860+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:20.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:19.860+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:20.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:19.861+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:21.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:21.117+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:21.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:21.117+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:21.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:21.117+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:21.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:21.118+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:20.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:21.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:20.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:21.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:20.894+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:21.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:20.894+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:21.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:20.900+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:21.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:20.900+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:21.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:20.900+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:21.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:20.900+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:22.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:21.883+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:22.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:21.883+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:22.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:21.883+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:22.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:21.884+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:22.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:22.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:22.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:22.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:22.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:22.120+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:22.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:22.121+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:22.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:21.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:22.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:21.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:22.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:21.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:22.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:21.906+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:23.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:43:23.087+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30274 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:43:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:22.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:22.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:22.950+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:23.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:22.950+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:23.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:22.859+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:23.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:22.859+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:23.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:22.859+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:23.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:22.860+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:23.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:23.105+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:23.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:23.105+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:23.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:23.105+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:23.333 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:23.106+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:23.982 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:23.851+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:23.982 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:23.851+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:23.983 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:23.851+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:23.983 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:23.851+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:24.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:23.980+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:23.980+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:23.980+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:24.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:23.981+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:24.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:24.331 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:24.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:24.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:24.127+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:24.332 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:24.127+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:24.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:43:24.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:43:25.168 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:24.877+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:25.169 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:24.877+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:25.169 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:24.877+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:25.169 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:24.878+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:25.169 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:24.934+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:25.169 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:24.934+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:25.169 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:24.934+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:25.170 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:24.935+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:25.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:25.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:25.166+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:25.167+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:26.194 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:25.910+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:26.195 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:25.910+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:26.195 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:25.910+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:26.195 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:25.911+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:26.195 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:25.850+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:26.195 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:25.850+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:26.195 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:25.850+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:26.195 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:25.851+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:26.192+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:26.192+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:26.192+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:26.193+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:26.816+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:26.816+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:26.816+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:27.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:26.816+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:27.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:26.892+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:27.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:26.892+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:27.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:26.892+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:27.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:26.893+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:27.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:27.145+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:27.145+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:27.145+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:27.145+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:27.773 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:43:28.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:27.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:28.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:27.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:28.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:27.821+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:28.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:27.822+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:28.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:27.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:28.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:27.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:28.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:27.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:28.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:27.876+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:28.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:28.110+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:28.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:28.110+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:28.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:28.110+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:28.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:28.110+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:28.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:43:28.091+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30282 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:43:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:28.855+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:28.855+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:28.855+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:29.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:28.856+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:29.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:28.885+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:29.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:28.885+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:29.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:28.885+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:29.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:28.886+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:29.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:29.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:29.149+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:29.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:29.150+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:30.153 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:29.913+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:30.153 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:29.913+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:30.154 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:29.913+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:30.154 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:29.914+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:30.154 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:29.873+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:30.154 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:29.873+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:30.154 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:29.873+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:30.154 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:29.874+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:30.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:30.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:30.151+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:30.152+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:31.183 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:30.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:31.184 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:30.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:31.184 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:30.869+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:31.184 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:30.870+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:31.184 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:30.915+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:31.184 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:30.915+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:31.184 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:30.915+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:31.185 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:30.916+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:31.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:31.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:31.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:31.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:31.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:31.181+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:31.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:31.181+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:32.221 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:31.963+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:32.221 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:31.963+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:32.222 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:31.963+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:32.222 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:31.964+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:32.222 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:31.873+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:32.222 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:31.873+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:32.222 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:31.873+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:32.222 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:31.874+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:32.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:32.219+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:32.219+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:32.219+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:32.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:32.220+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:33.255 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:43:33.094+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30289 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:43:33.256 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:33.011+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:33.256 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:33.011+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:33.256 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:33.011+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:33.256 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:33.012+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:33.256 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:32.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:33.256 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:32.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:33.257 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:32.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:33.257 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:32.923+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:33.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:33.253+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:33.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:33.253+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:33.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:33.253+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:33.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:33.254+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:33.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:33.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:33.895+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:33.895+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:34.083 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:33.978+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:34.084 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:33.978+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:34.084 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:33.978+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:34.084 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:33.979+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:34.280+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:34.280+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:34.280+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:34.281+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:35.318 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:34.847+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:35.318 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:34.847+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:35.318 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:34.847+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:35.319 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:34.847+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:35.319 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:34.999+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:35.319 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:34.999+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:35.319 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:34.999+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:35.319 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:35.000+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:35.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:35.316+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:35.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:35.316+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:35.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:35.316+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:35.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:35.317+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:36.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:35.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:36.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:35.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:36.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:35.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:36.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:35.828+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:35.970+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:36.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:35.970+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:36.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:35.970+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:36.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:35.970+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:36.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:36.302+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:36.302+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:36.302+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:36.303+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:37.264 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:37.017+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:37.264 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:37.017+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:37.264 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:37.017+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:37.264 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:37.017+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:37.265 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:36.841+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:37.265 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:36.841+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:37.265 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:36.841+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:37.265 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:36.842+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:37.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:37.262+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:37.262+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:37.262+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:37.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:37.263+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:38.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:38.023+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:38.023+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:38.023+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:38.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:38.024+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:38.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:37.824+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:38.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:37.824+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:38.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:37.824+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:38.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:37.825+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:38.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:43:38.097+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30297 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:43:38.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:38.225+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:38.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:38.225+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:38.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:38.225+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:38.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:38.225+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:38.814+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:38.814+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:38.814+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:38.814+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:39.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:39.061+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:39.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:39.061+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:39.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:39.061+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:39.082 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:39.062+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:39.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:39.253+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:39.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:39.253+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:39.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:39.253+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:39.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:39.254+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:39.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:43:39.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:43:40.259 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:40.055+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:40.259 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:40.055+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:40.259 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:40.055+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:40.260 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:40.055+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:40.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:39.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:40.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:39.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:40.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:39.832+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:40.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:39.833+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:40.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:40.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:40.257+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:40.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:40.258+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:41.275 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:41.038+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:41.275 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:41.038+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:41.275 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:41.038+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:41.275 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:41.038+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:41.276 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:40.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:41.276 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:40.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:41.276 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:40.871+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:41.276 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:40.872+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:41.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:41.273+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:41.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:41.273+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:41.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:41.273+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:41.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:41.274+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:42.285 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:42.003+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:42.285 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:42.003+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:42.286 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:42.003+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:42.286 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:42.004+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:42.286 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:41.837+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:42.286 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:41.837+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:42.286 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:41.837+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:42.286 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:41.838+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:42.283+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:42.283+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:42.283+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:42.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:42.284+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:42.845 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:43:43.306 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:42.861+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:43.306 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:42.861+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:43.306 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:42.861+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:43.306 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:42.861+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:43.307 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:42.974+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:43.307 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:42.974+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:43.307 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:42.974+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:43.307 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:42.974+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:43.308 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:43:43.100+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30304 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:43:43.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:43.304+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:43.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:43.304+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:43.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:43.304+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:43.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:43.305+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:44.064 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:43.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:44.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:43.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:44.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:43.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:44.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:43.906+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:44.065 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:43.994+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:44.065 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:43.994+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:44.065 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:43.994+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:44.066 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:43.995+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:44.598 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:44.337+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:44.598 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:44.337+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:44.599 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:44.337+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:44.599 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:44.338+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:45.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:44.971+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:44.971+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:44.971+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:45.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:44.972+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:44.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:45.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:44.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:45.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:44.879+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:45.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:44.880+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:45.696 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:45.337+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:45.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:45.337+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:45.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:45.337+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:45.697 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:45.338+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:45.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:43:45.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:43:45.921 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:43:45.921 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-10T04:43:45.922 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-10T04:43:45.922 DEBUG:teuthology.orchestra.run.smithi112:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-10T04:43:45.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:43:45.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:43:46.006 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:45.835+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:46.006 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:45.835+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:46.006 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:45.835+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:46.007 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:45.836+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:46.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:46.003+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:46.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:46.003+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:46.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:46.003+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:46.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:46.004+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:46.387+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:46.387+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:46.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:46.387+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:46.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:46.388+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:47.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:47.013+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:47.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:47.013+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:47.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:47.013+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:47.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:47.014+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:47.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:46.829+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:47.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:46.829+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:47.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:46.829+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:47.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:46.830+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:47.807 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:47.350+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:47.807 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:47.350+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:47.808 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:47.350+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:47.808 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:47.350+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:48.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:47.999+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:47.999+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:47.999+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:48.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:48.000+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:47.805+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:47.805+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:48.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:47.805+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:48.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:47.806+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:48.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:48.348+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:48.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:48.348+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:48.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:48.348+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:48.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:48.349+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:48.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:43:48.103+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30312 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:43:49.025 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:43:49.198 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:49.022+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:49.198 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:49.022+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:49.199 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:49.022+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:49.199 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:49.023+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:49.199 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:48.830+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:49.199 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:48.830+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:49.199 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:48.830+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:49.200 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:48.831+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:49.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:49.390+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:49.390+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:49.390+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:49.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:49.391+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:50.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:50.024+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:50.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:50.024+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:50.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:50.024+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:50.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:50.025+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:50.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:49.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:50.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:49.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:50.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:49.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:50.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:49.863+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:50.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:50.350+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:50.350+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:50.350+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:50.350+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:51.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:51.065+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:51.065+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:51.065+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:51.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:51.066+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:51.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:50.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:51.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:50.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:51.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:50.899+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:51.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:50.900+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:51.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:51.344+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:51.344+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:51.344+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:51.344+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:52.068+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:52.068+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:52.068+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:52.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:52.069+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:52.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:51.911+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:52.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:51.911+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:52.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:51.911+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:52.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:51.912+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:52.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:52.347+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:52.347+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:52.347+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:52.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:52.347+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:52.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:52.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:52.922+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:53.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:52.923+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:53.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:53.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:53.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:53.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:53.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:53.073+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:53.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:53.074+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:53.332 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:43:53.106+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30319 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:43:53.764 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:53.356+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:53.764 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:53.356+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:53.764 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:53.356+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:53.764 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:53.357+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:54.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:53.886+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:53.886+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:53.886+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:53.887+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:54.374 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:54.083+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:54.375 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:54.083+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:54.375 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:54.083+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:54.375 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:54.083+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:54.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:54.373+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:54.373+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:54.373+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:54.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:54.373+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:55.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:55.105+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:55.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:55.105+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:55.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:55.105+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:55.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:55.106+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:55.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:54.924+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:55.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:54.924+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:55.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:54.924+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:55.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:54.924+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:55.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:55.336+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:55.336+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:55.336+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:55.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:55.337+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:56.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:56.092+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:56.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:56.092+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:56.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:56.092+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:56.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:56.093+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:55.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:56.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:55.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:56.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:55.930+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:56.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:55.931+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:56.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:56.374+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:56.374+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:56.374+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:56.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:56.375+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:57.144 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:56.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:57.144 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:56.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:57.144 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:56.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:57.144 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:56.971+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:57.530 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:57.141+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:57.531 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:57.141+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:57.531 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:57.141+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:57.531 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:57.142+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:57.531 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:57.344+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:57.532 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:57.344+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:57.532 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:57.344+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:57.532 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:57.345+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:58.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:43:58.027 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:43:58.326 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:43:58.109+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30327 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:43:58.327 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:58.005+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:58.327 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:58.005+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:58.327 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:58.005+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:58.327 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:58.005+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:58.327 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:58.099+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:58.327 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:58.099+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:58.328 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:58.099+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:58.328 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:58.099+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:58.323+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:58.323+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:58.323+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:58.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:58.324+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:59.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:59.059+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:59.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:59.059+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:59.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:59.059+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:43:59.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:43:59.060+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:43:59.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:58.986+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:59.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:58.986+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:59.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:58.986+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:43:59.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:58.987+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:43:59.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:59.368+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:59.368+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:59.368+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:43:59.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:43:59.368+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:00.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:00.077+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:00.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:00.077+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:00.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:00.077+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:00.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:00.078+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:59.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:00.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:59.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:00.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:59.966+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:00.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:43:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:43:59.966+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:00.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:00.406+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:00.406+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:00.406+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:00.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:00.407+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:01.085 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:44:01.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:01.030+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:01.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:01.030+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:01.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:01.030+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:01.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:01.031+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:01.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:00.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:01.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:00.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:01.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:00.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:01.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:00.979+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:01.444+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:01.444+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:01.444+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:01.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:01.445+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:02.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:02.074+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:02.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:02.074+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:02.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:02.074+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:02.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:02.074+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:01.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:01.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:02.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:01.957+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:02.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:01.958+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:02.430+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:02.430+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:02.430+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:02.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:02.431+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:03.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:44:03.113+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30334 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:44:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:02.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:02.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:03.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:02.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:03.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:02.953+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:03.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:03.064+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:03.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:03.064+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:03.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:03.064+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:03.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:03.065+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:03.764 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:03.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:03.765 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:03.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:03.765 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:03.407+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:03.765 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:03.408+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:04.021 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:03.944+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:04.021 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:03.944+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:04.021 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:03.944+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:04.022 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:03.944+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:04.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:04.019+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:04.019+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:04.019+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:04.020+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:04.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:04.367+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:04.367+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:04.367+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:04.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:04.368+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:05.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:05.064+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:05.064+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:05.064+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:05.064+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:04.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:05.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:04.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:05.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:04.978+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:05.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:04.979+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:05.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:05.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:05.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:05.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:05.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:05.403+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:06.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:05.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:05.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:05.971+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:06.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:05.972+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:06.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:06.105+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:06.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:06.105+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:06.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:06.105+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:06.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:06.106+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:06.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:06.413+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:06.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:06.413+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:06.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:06.413+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:06.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:06.413+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:07.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:07.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:07.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:07.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:07.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:07.002+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:07.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:07.003+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:07.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:07.112+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:07.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:07.112+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:07.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:07.112+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:07.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:07.113+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:07.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:07.403+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:07.403+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:07.403+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:07.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:07.403+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:08.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:44:08.116+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30342 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:44:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:08.125+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:08.125+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:08.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:08.125+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:08.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:08.126+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:08.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:08.007+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:08.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:08.007+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:08.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:08.007+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:08.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:08.007+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:08.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:08.388+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:08.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:08.388+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:08.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:08.388+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:08.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:08.389+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:09.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:09.130+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:09.130+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:09.130+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:09.131+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:09.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:09.020+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:09.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:09.020+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:09.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:09.020+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:09.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:09.020+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:09.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:09.400+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:09.400+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:09.400+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:09.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:09.401+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:10.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:10.099+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:10.099+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:10.099+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:10.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:10.099+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:10.006+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:10.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:10.006+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:10.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:10.006+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:10.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:10.007+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:10.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:10.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:10.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:10.389+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:10.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:10.389+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:11.096+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:11.096+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:11.096+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:11.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:11.096+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:11.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:11.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:11.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:11.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:11.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:11.031+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:11.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:11.032+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:11.376+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:11.376+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:11.376+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:11.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:11.377+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:12.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:11.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:12.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:11.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:12.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:11.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:12.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:11.999+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:12.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:12.101+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:12.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:12.101+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:12.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:12.101+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:12.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:12.102+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:12.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:12.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:12.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:12.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:12.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:12.403+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:13.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:44:13.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:44:13.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:44:13.119+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30349 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:44:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:12.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:12.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:12.999+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:13.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:13.000+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:13.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:13.063+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:13.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:13.063+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:13.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:13.063+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:13.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:13.064+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:13.765 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:13.442+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:13.765 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:13.442+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:13.765 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:13.442+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:13.765 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:13.442+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:14.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:13.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:13.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:13.973+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:13.974+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:14.443 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:14.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:14.443 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:14.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:14.444 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:14.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:14.444 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:14.098+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:14.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:14.442+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:14.442+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:14.442+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:14.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:14.442+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:15.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:14.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:14.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:14.941+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:15.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:14.942+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:15.066+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:15.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:15.066+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:15.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:15.066+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:15.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:15.067+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:15.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:15.400+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:15.400+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:15.400+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:15.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:15.401+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:16.157 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:44:16.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:15.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:16.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:15.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:16.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:15.943+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:16.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:15.944+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:16.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:16.105+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:16.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:16.105+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:16.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:16.105+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:16.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:16.106+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:16.736 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:16.432+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:16.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:16.432+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:16.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:16.432+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:16.737 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:16.433+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:17.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:16.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:16.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:16.919+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:17.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:16.920+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:17.442 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:17.129+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:17.442 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:17.129+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:17.443 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:17.129+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:17.443 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:17.130+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:17.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:17.440+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:17.440+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:17.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:17.440+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:17.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:17.441+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:18.330 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:44:18.122+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30357 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:44:18.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:17.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:18.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:17.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:18.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:17.904+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:18.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:17.905+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:18.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:18.128+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:18.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:18.128+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:18.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:18.128+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:18.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:18.129+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:18.478+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:18.478+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:18.478+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:18.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:18.479+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:19.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:44:19.158 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:44:19.233 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:44:19.233 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-10T04:44:19.233 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-10T04:44:19.236 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/adking-2024-09-10_02:59:23-orch:cephadm-wip-adk-testing-2024-09-09-1904-distro-default-smithi/7898573/remote/smithi112/crash 2024-09-10T04:44:19.237 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/crash -- . 2024-09-10T04:44:19.259 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:19.085+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:19.259 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:19.085+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:19.259 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:19.085+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:19.259 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:19.086+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:19.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:18.915+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:19.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:18.915+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:19.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:18.915+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:19.260 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:18.916+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:19.264 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/adking-2024-09-10_02:59:23-orch:cephadm-wip-adk-testing-2024-09-09-1904-distro-default-smithi/7898573/remote/smithi169/crash 2024-09-10T04:44:19.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:44:19.265 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:44:19.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:19.435+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:19.435+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:19.435+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:19.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:19.435+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:20.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:19.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:19.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:19.952+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:20.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:19.953+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:20.399 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:20.128+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:20.399 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:20.128+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:20.400 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:20.128+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:20.400 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:20.129+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:20.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:20.398+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:20.398+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:20.398+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:20.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:20.398+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:21.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:20.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:20.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:20.905+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:21.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:20.906+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:21.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:21.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:21.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:21.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:21.133+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:21.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:21.401+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:21.401+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:21.401+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:21.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:21.401+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:22.147 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:21.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:22.147 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:21.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:22.147 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:21.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:22.148 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:21.858+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:22.333 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:44:22.436 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:22.145+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:22.437 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:22.145+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:22.437 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:22.145+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:22.437 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:22.146+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:22.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:22.434+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:22.435+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:22.435+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:22.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:22.435+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:23.127 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:22.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:23.128 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:22.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:23.128 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:22.857+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:23.128 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:22.858+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:23.417 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:44:23.125+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30364 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:44:23.418 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:23.176+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:23.418 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:23.176+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:23.418 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:23.176+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:23.418 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:23.177+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:23.765 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:23.416+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:23.765 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:23.416+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:23.765 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:23.416+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:23.765 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:23.416+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:24.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:23.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:24.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:23.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:24.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:23.893+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:24.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:23.894+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:24.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:24.432+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:24.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:24.432+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:24.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:24.432+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:24.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:24.432+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:24.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:24.194+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:24.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:24.194+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:24.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:24.194+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:24.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:24.195+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:25.229 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:24.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:25.229 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:24.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:25.229 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:24.890+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:25.229 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:24.891+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:25.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:25.227+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:25.227+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:25.227+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:25.228+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:25.391+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:25.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:25.391+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:25.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:25.391+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:25.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:25.392+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:26.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:25.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:26.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:25.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:26.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:25.862+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:26.255 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:25.862+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:26.362+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:26.362+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:26.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:26.362+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:26.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:26.363+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:26.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:26.253+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:26.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:26.253+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:26.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:26.253+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:26.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:26.254+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:27.273 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:26.859+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:27.273 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:26.859+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:27.274 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:26.859+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:27.274 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:26.860+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:27.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:27.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:27.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:27.271+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:27.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:27.272+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:27.388+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:27.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:27.388+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:27.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:27.388+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:27.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:27.389+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:28.130 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:27.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:28.130 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:27.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:28.131 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:27.870+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:28.131 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:27.871+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:28.381 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:28.245+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:28.382 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:28.245+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:28.382 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:28.245+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:28.382 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:28.246+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:28.382 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:44:28.128+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30372 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:44:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:28.380+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:28.380+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:28.380+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:28.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:28.380+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:29.206 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:28.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:29.207 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:28.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:29.207 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:28.875+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:29.207 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:28.876+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:29.400+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:29.400+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:29.400+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:29.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:29.400+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:29.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:29.205+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:29.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:29.205+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:29.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:29.205+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:29.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:29.205+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:30.170 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:29.901+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:30.171 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:29.901+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:30.171 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:29.901+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:30.171 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:29.902+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:30.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:30.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:30.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:30.409+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:30.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:30.410+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:30.169+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:30.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:30.169+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:30.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:30.169+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:30.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:30.169+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:31.126 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:30.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:31.127 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:30.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:31.127 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:30.855+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:31.127 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:30.856+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:31.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:44:31.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:44:31.416 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:31.125+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:31.416 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:31.125+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:31.416 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:31.125+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:31.417 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:31.126+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:31.810 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:31.414+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:31.810 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:31.414+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:31.810 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:31.414+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:31.810 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:31.414+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:32.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:31.808+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:31.808+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:31.808+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:32.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:31.809+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:32.444 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:32.166+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:32.445 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:32.166+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:32.445 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:32.166+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:32.445 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:32.167+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:32.786 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:32.441+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:32.786 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:32.441+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:32.787 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:32.441+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:32.787 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:32.442+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:33.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:32.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:33.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:32.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:33.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:32.783+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:33.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:32.784+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:33.423 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:33.184+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:33.424 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:33.184+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:33.424 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:33.184+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:33.424 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:33.185+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:33.424 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:44:33.130+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30379 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:44:33.765 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:33.421+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:33.766 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:33.421+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:33.766 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:33.421+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:33.766 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:33.422+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:33.813+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:33.813+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:33.813+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:33.814+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:34.397 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:44:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:34.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:34.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:34.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:34.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:34.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:34.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:34.468+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:34.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:34.235+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:34.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:34.235+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:34.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:34.235+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:34.583 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:34.235+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:35.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:34.817+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:35.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:34.817+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:35.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:34.817+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:35.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:34.818+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:35.441 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:35.215+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:35.441 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:35.215+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:35.442 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:35.215+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:35.442 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:35.216+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:35.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:35.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:35.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:35.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:35.440+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:36.206 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:35.834+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:36.206 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:35.834+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:36.207 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:35.834+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:36.207 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:35.835+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:36.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:36.203+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:36.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:36.203+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:36.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:36.203+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:36.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:36.204+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:36.436+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:36.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:36.436+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:36.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:36.436+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:36.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:36.436+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:37.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:36.819+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:36.819+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:36.819+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:36.820+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:37.434 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:37.162+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:37.434 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:37.162+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:37.435 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:37.162+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:37.435 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:37.162+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:37.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:37.431+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:37.431+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:37.431+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:37.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:37.432+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:38.136 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:37.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:38.137 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:37.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:38.137 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:37.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:38.137 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:37.853+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:38.469 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:44:38.133+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30387 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:44:38.469 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:38.169+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:38.470 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:38.169+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:38.470 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:38.169+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:38.470 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:38.169+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:38.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:38.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:38.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:38.467+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:38.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:38.468+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:39.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:39.121+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:39.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:39.121+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:39.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:39.121+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:39.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:39.122+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:39.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:38.888+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:39.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:38.888+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:39.332 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:38.888+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:39.333 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:38.889+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:39.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:39.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:39.493+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:39.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:39.494+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:40.131 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:39.848+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:40.131 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:39.848+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:40.132 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:39.848+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:40.132 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:39.848+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:40.450 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:40.130+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:40.451 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:40.130+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:40.451 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:40.130+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:40.451 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:40.130+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:40.449+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:40.449+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:40.449+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:40.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:40.449+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:41.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:40.828+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:41.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:40.828+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:41.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:40.828+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:41.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:40.829+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:41.404 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:41.120+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:41.405 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:41.120+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:41.405 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:41.120+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:41.405 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:41.120+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:41.797 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:41.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:41.798 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:41.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:41.798 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:41.402+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:41.798 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:41.403+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:42.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:41.795+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:42.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:41.795+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:42.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:41.795+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:42.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:41.796+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:42.145+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:42.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:42.145+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:42.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:42.145+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:42.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:42.146+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:42.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:42.367+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:42.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:42.367+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:42.583 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:42.367+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:42.583 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:42.368+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:42.793+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:42.793+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:42.793+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:42.794+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:43.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:44:43.138+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30394 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:44:43.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:43.357+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:43.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:43.357+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:43.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:43.357+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:43.582 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:43.358+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:43.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:43.155+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:43.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:43.155+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:43.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:43.155+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:43.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:43.156+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:44.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:43.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:44.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:43.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:44.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:43.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:44.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:43.766+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:44.385 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:44.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:44.385 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:44.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:44.385 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:44.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:44.385 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:44.120+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:44.752 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:44.391+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:44.753 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:44.391+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:44.753 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:44.391+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:44.753 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:44.392+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:45.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:44.751+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:45.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:44.751+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:45.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:44.751+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:45.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:44.752+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:45.441 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:45.140+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:45.442 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:45.140+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:45.442 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:45.140+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:45.442 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:45.141+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:45.783 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:45.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:45.784 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:45.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:45.784 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:45.439+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:45.784 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:45.440+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:46.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:45.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:46.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:45.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:46.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:45.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:46.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:45.782+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:46.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:44:46.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:44:46.485 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:46.187+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:46.486 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:46.187+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:46.486 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:46.187+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:46.486 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:46.188+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:46.806 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:46.484+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:46.806 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:46.484+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:46.807 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:46.484+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:46.807 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:46.485+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:47.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:46.805+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:46.805+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:46.805+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:46.805+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:47.509 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:47.212+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:47.509 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:47.212+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:47.509 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:47.212+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:47.509 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:47.213+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:47.785 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:47.507+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:47.786 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:47.507+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:47.786 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:47.507+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:47.786 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:47.508+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:47.784+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:47.784+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:47.784+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:47.785+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:48.529 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:44:48.141+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30402 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:44:48.529 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:48.163+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:48.530 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:48.163+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:48.530 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:48.163+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:48.530 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:48.164+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:48.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:48.756+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:48.756+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:48.756+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:48.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:48.756+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:48.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:48.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:48.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:48.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:48.527+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:48.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:48.528+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:49.473 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:44:49.513 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:49.144+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:49.513 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:49.144+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:49.513 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:49.144+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:49.514 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:49.145+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:49.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:49.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:49.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:49.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:49.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:49.755+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:49.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:49.511+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:49.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:49.511+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:49.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:49.511+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:49.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:49.512+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:50.522 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:50.175+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:50.522 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:50.175+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:50.523 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:50.175+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:50.523 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:50.176+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:50.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:50.723+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:50.723+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:50.723+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:50.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:50.724+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:50.520+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:50.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:50.520+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:50.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:50.520+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:50.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:50.521+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:51.507 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:51.210+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:51.508 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:51.210+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:51.508 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:51.210+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:51.508 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:51.211+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:51.830 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:51.506+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:51.506+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:51.506+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:51.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:51.506+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:51.691+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:51.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:51.691+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:51.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:51.691+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:51.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:51.692+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:52.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:44:52.474 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:44:52.541 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:44:52.541 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-10T04:44:52.541 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-10T04:44:52.542 DEBUG:teuthology.orchestra.run.smithi112:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-09-10T04:44:52.552 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:52.231+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:52.552 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:52.231+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:52.552 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:52.231+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:52.552 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:52.232+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:52.586 INFO:tasks.cephadm:Compressing logs... 2024-09-10T04:44:52.586 DEBUG:teuthology.orchestra.run.smithi112:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-10T04:44:52.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:44:52.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:44:52.654 INFO:teuthology.orchestra.run.smithi112.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-osd.0.log 2024-09-10T04:44:52.654 INFO:teuthology.orchestra.run.smithi112.stderr:: No such file or directory 2024-09-10T04:44:52.654 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-osd.1.log 2024-09-10T04:44:52.654 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-mgr.smithi112.pqrlrb.log 2024-09-10T04:44:52.656 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-osd.0.log: /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-10T04:44:52.656 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-mgr.smithi112.pqrlrb.log: gzip -5 --verbose -- /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph.audit.log 2024-09-10T04:44:52.656 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-mds.cephfs.smithi112.uuenbo.log 2024-09-10T04:44:52.657 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-mds.cephfs.smithi112.ifnhnl.log 2024-09-10T04:44:52.658 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph.audit.log: /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-mds.cephfs.smithi112.uuenbo.log: 90.7% -- replaced with /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph.audit.log.gz 2024-09-10T04:44:52.667 INFO:teuthology.orchestra.run.smithi112.stderr: 90.7% -- replaced with /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-10T04:44:52.668 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph.cephadm.log 2024-09-10T04:44:52.668 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-mds.cephfs.smithi112.ifnhnl.log: gzip -5 --verbose -- /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-osd.2.log 2024-09-10T04:44:52.669 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph.cephadm.log: 84.4% -- replaced with /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph.cephadm.log.gz 2024-09-10T04:44:52.669 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph.log 2024-09-10T04:44:52.676 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-client.ceph-exporter.smithi112.log 2024-09-10T04:44:52.677 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph.log: 87.0% -- replaced with /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph.log.gz 2024-09-10T04:44:52.690 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-mon.smithi112.log 2024-09-10T04:44:52.697 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-client.ceph-exporter.smithi112.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-10T04:44:52.697 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-mon.smithi112.log: 94.7% -- replaced with /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-client.ceph-exporter.smithi112.log.gz 2024-09-10T04:44:52.723 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cephadm.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-10T04:44:52.754 INFO:teuthology.orchestra.run.smithi112.stderr: 95.3% -- replaced with /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-mds.cephfs.smithi112.ifnhnl.log.gz 2024-09-10T04:44:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:52.643+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:52.643+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:52.643+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:52.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:52.643+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:52.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:52.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:52.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:52.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:52.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:52.550+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:52.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:52.551+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:52.921 INFO:teuthology.orchestra.run.smithi112.stderr: 92.0% -- replaced with /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-mgr.smithi112.pqrlrb.log.gz 2024-09-10T04:44:53.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:44:53.144+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30409 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:44:53.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:53.249+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:53.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:53.249+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:53.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:53.249+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:53.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:53.249+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:53.673+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:53.673+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:53.673+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:53.673+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:54.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:53.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:54.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:53.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:54.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:53.596+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:54.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:53.596+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:54.224+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:54.224+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:54.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:54.224+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:54.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:54.224+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:55.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:54.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:55.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:54.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:55.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:54.586+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:55.083 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:54.586+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:55.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:54.688+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:55.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:54.688+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:55.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:54.688+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:55.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:54.688+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:55.457 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:55.190+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:55.457 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:55.190+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:55.457 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:55.190+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:55.458 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:55.190+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:55.713 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:44:55.726 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:55.565+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:55.726 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:55.565+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:55.726 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:55.565+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:55.726 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:55.565+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:55.726 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:55.722+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:55.726 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:55.722+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:55.726 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:55.722+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:56.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:55.722+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:56.162+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:56.162+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:56.162+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:56.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:56.162+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:56.603+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:56.603+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:56.603+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:57.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:56.604+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:57.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:56.685+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:57.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:56.685+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:57.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:56.685+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:57.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:56.687+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:57.154+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:57.154+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:57.154+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:57.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:57.155+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:57.598+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:57.598+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:57.598+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:58.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:57.598+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:58.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:57.663+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:58.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:57.663+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:58.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:57.663+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:58.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:57.664+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:58.153+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:58.153+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:58.153+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:58.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:58.153+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:58.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:44:58.149+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30417 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:44:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:58.577+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:58.577+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:58.577+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:58.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:58.578+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:58.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:58.701+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:58.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:58.701+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:58.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:58.701+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:58.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:58.701+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:59.565 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:59.562+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:59.566 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:59.562+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:59.566 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:59.562+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:44:59.566 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:44:59.563+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:44:59.566 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:59.131+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:59.566 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:59.131+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:59.566 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:59.131+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:44:59.567 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:44:59.132+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:44:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:59.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:59.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:59.668+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:44:59.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:44:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:44:59.668+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:00.554 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:00.552+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:00.555 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:00.552+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:00.555 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:00.552+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:00.555 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:00.117+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:00.555 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:00.117+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:00.556 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:00.117+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:00.556 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:00.117+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:00.713+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:00.713+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:00.713+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:00.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:00.713+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:00.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:00.552+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:00.937 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-mon.smithi112.log: file size changed while zipping 2024-09-10T04:45:00.938 INFO:teuthology.orchestra.run.smithi112.stderr: 98.0% -- replaced with /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-mon.smithi112.log.gz 2024-09-10T04:45:01.559 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:01.090+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:01.559 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:01.090+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:01.559 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:01.090+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:01.559 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:01.090+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:01.560 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:01.556+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:01.560 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:01.556+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:01.560 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:01.556+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:01.674+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:01.674+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:01.674+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:01.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:01.675+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:01.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:01.557+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:02.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:02.072+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:02.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:02.072+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:02.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:02.072+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:02.333 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:02.072+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:02.703+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:02.703+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:02.703+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:02.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:02.703+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:02.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:02.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:02.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:02.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:02.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:02.549+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:02.832 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:02.549+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:03.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:45:03.153+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30424 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:45:03.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:03.037+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:03.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:03.037+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:03.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:03.037+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:03.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:03.038+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:03.378 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-osd.2.log: file size changed while zipping 2024-09-10T04:45:03.378 INFO:teuthology.orchestra.run.smithi112.stderr: 95.0% -- replaced with /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-osd.2.log.gz 2024-09-10T04:45:04.059 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:03.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:04.059 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:03.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:04.059 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:03.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:04.059 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:03.591+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:04.060 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:04.057+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:04.060 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:04.057+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:04.060 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:04.057+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:04.060 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:03.729+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:04.060 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:03.729+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:04.061 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:03.729+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:04.061 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:03 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:03.729+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:04.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:04.058+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:04.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:45:04.715 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:45:05.054 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:04.594+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:05.055 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:04.594+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:05.055 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:04.594+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:05.055 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:04.594+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:05.056 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:04.721+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:05.056 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:04.721+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:05.056 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:04.721+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:05.057 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:04 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:04.721+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:05.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:05.052+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:05.052+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:05.052+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:05.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:05.053+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:05.954 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:05.603+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:05.954 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:05.603+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:05.954 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:05.603+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:05.954 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:05.603+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:05.955 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:05.720+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:05.955 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:05.720+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:05.955 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:05.720+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:05.955 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:05 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:05.720+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:06.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:06.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:06.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:06.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:06.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:06.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:06.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:06.098+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:07.053 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:06.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:07.053 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:06.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:07.053 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:06.591+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:07.053 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:06.592+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:07.054 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:06.683+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:07.054 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:06.683+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:07.054 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:06.683+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:07.054 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:06 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:06.683+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:07.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:07.051+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:07.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:07.051+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:07.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:07.051+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:07.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:07.052+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:07.773 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:45:07.998 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-osd.0.log: file size changed while zipping 2024-09-10T04:45:07.998 INFO:teuthology.orchestra.run.smithi112.stderr: 96.4% -- replaced with /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-osd.0.log.gz 2024-09-10T04:45:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:07.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:07.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:07.626+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:08.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:07.627+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:08.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:07.713+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:08.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:07.713+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:08.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:07.713+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:08.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:07 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:07.714+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:08.144 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-osd.1.log: file size changed while zipping 2024-09-10T04:45:08.145 INFO:teuthology.orchestra.run.smithi112.stderr: 96.4% -- replaced with /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-osd.1.log.gz 2024-09-10T04:45:08.580 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:45:08.156+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30432 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:45:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:08.100+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:08.100+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:08.100+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:08.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:08.101+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:09.062 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:08.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:09.062 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:08.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:09.063 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:08.648+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:09.063 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:08.649+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:09.063 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:08.695+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:09.063 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:08.695+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:09.063 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:08.695+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:09.064 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:08 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:08.696+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:09.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:09.060+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:09.060+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:09.060+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:09.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:09.061+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:10.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:09.607+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:09.607+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:09.607+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:10.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:09.608+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:10.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:09.715+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:10.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:09.715+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:10.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:09.715+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:10.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:09 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:09.716+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:10.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:10.099+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:10.099+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:10.099+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:10.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:10.099+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:10.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:10.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:10.650+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:11.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:10.650+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:11.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:10.703+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:11.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:10.703+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:11.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:10.703+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:11.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:10 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:10.704+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:11.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:11.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:11.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:11.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:11.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:11.133+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:12.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:11.683+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:11.684+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:11.684+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:12.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:11.684+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:12.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:11.694+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:12.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:11.694+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:12.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:11.694+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:12.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:11 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:11.695+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:12.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:12.159+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:12.159+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:12.159+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:12.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:12.160+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:13.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:12.709+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:13.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:12.709+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:13.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:12.709+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:13.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:12.710+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:13.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:12.699+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:13.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:12.699+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:13.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:12.699+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:13.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:12 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:12.700+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:13.581 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:45:13.158+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30439 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:45:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:13.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:13.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:13.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:13.151+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:13.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:13.152+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:14.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:13.700+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:13.700+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:13.700+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:14.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:13.702+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:13.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:14.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:13.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:14.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:13.709+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:14.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:13 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:13.710+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:14.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:14.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:14.132+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:14.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:14.133+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:15.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:14.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:14.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:14.722+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:15.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:14.723+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:15.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:14.718+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:15.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:14.718+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:15.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:14.718+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:15.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:14 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:14.719+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:15.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:15.123+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:15.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:15.123+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:15.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:15.123+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:15.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:15.124+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:15.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:15.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:15.772+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:16.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:15.773+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:16.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:15.725+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:16.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:15.725+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:16.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:15.725+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:16.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:15 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:15.726+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:16.156+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:16.156+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:16.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:16.156+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:16.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:16.156+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:16.792+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:16.792+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:16.792+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:17.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:16.792+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:17.084 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:16.694+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:17.084 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:16.694+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:17.084 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:16.694+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:17.084 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:16 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:16.695+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:17.111+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:17.111+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:17.111+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:17.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:17.112+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:18.077 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:17.688+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:18.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:17.688+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:18.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:17.688+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:18.078 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:17.689+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:18.078 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:17.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:18.078 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:17.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:18.079 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:17.837+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:18.079 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:17 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:17.838+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:18.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:18.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:18.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:18.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:18.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:18.075+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:18.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:18.076+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:18.331 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:45:18.162+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30447 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:45:19.046 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:18.827+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:19.046 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:18.827+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:19.046 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:18.827+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:19.047 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:18.827+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:19.047 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:18.654+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:19.047 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:18.654+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:19.047 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:18.654+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:19.047 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:18 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:18.655+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:19.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:19.044+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:19.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:19.044+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:19.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:19.044+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:19.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:19.045+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:19.773 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:45:19.774 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:45:20.068 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:19.783+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:20.068 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:19.783+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:20.068 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:19.783+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:20.069 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:19.783+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:20.069 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:19.662+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:20.069 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:19.662+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:20.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:19.662+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:20.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:19 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:19.662+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:20.066+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:20.066+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:20.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:20.066+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:20.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:20.066+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:21.062 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:20.704+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:21.062 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:20.704+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:21.063 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:20.704+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:21.063 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:20.705+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:21.063 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:20.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:21.063 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:20.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:21.063 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:20.768+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:21.063 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:20 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:20.769+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:21.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:21.060+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:21.060+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:21.060+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:21.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:21.061+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:21.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:21.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:21.803+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:22.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:21.804+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:22.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:21.699+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:22.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:21.699+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:22.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:21.699+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:22.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:21 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:21.700+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:22.081+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:22.081+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:22.081+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:22.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:22.082+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:22.849 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:45:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:22.807+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:22.807+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:22.807+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:23.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:22.808+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:23.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:22.735+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:23.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:22.735+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:23.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:22.735+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:23.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:22 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:22.736+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:23.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:23.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:23.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:23.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:23.097+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:23.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:23.098+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:23.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:45:23.166+0000 7fd3b2712640 -1 mon.smithi112@0(probing) e2 get_health_metrics reporting 30454 slow ops, oldest is mgrbeacon mgr.smithi112.pqrlrb(513b3bc6-6f25-11ef-bcea-c7b262605968,14680, , 0) 2024-09-10T04:45:24.063 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:23.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:24.064 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:23.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:24.064 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:23.820+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:24.064 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:23.820+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:24.064 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:23.711+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:24.064 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:23.711+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:24.064 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:23.711+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:24.065 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:23 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:23.712+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:24.061+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:24.061+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:24.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:24.061+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:24.332 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:24.062+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:24.813+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:24.813+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:24.813+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:25.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:24.814+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:25.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:24.727+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:25.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:24.727+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:25.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:24.727+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:25.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:24 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:24.728+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:25.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:25.096+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:25.096+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:25.096+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:25.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:25.096+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:25.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:45:25.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:45:25.921 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:45:25.921 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-10T04:45:25.921 INFO:tasks.cephadm:Removing cluster... 2024-09-10T04:45:25.922 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 --force 2024-09-10T04:45:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:25.794+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:25.794+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:26.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:25.794+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:26.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:25.795+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:26.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:25.732+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:26.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:25.732+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:26.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:25.732+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:26.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:25 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:25.734+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:26.159 INFO:teuthology.orchestra.run.smithi112.stdout:Deleting cluster with fsid: 513b3bc6-6f25-11ef-bcea-c7b262605968 2024-09-10T04:45:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:26.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:26.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:26.119+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:26.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:26.119+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:26.582 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:45:26 smithi112 systemd[1]: Stopping Ceph mon.smithi112 for 513b3bc6-6f25-11ef-bcea-c7b262605968... 2024-09-10T04:45:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:26.831+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:26.831+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:26.831+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:27.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:26.832+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:27.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:26.704+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:27.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:26.704+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:27.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:26.704+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:27.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:26.705+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:27.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:45:26.636+0000 7fd3b5718640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi112 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T04:45:27.082 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:45:26 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112[27733]: 2024-09-10T04:45:26.636+0000 7fd3b5718640 -1 mon.smithi112@0(probing) e2 *** Got Signal Terminated *** 2024-09-10T04:45:27.435 INFO:teuthology.orchestra.run.smithi112.stderr: 92.6% -- replaced with /var/log/ceph/513b3bc6-6f25-11ef-bcea-c7b262605968/ceph-mds.cephfs.smithi112.uuenbo.log.gz 2024-09-10T04:45:27.436 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:45:27 smithi112 podman[80399]: 2024-09-10 04:45:27.249289851 +0000 UTC m=+0.763780621 container died 91597bdf09f78a2afbdfa498aa1f0ab74554b1e43940ff53c2f6a2020ac8687e (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-b53b6d3, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T04:45:27.437 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:27.160+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:27.437 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:27.160+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:27.437 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:27.160+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:27.437 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:27.161+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:27.438 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-10T04:45:27.438 INFO:teuthology.orchestra.run.smithi112.stderr:real 0m34.796s 2024-09-10T04:45:27.438 INFO:teuthology.orchestra.run.smithi112.stderr:user 1m22.570s 2024-09-10T04:45:27.439 INFO:teuthology.orchestra.run.smithi112.stderr:sys 0m2.063s 2024-09-10T04:45:28.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:27.819+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:28.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:27.819+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:28.033 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:27.819+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:28.034 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:27.820+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:28.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:27.749+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:28.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:27.749+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:28.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:27.749+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:28.034 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:27 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:27.750+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:28.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:45:27 smithi112 podman[80399]: 2024-09-10 04:45:27.780660868 +0000 UTC m=+1.295151641 container cleanup 91597bdf09f78a2afbdfa498aa1f0ab74554b1e43940ff53c2f6a2020ac8687e (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-b53b6d3, CEPH_POINT_RELEASE=) 2024-09-10T04:45:28.035 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:45:27 smithi112 bash[80399]: ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112 2024-09-10T04:45:28.298 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:28.171+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:28.298 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:28.171+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:28.299 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:28.171+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:28.299 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:28.172+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:28.299 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:45:28 smithi112 podman[80410]: 2024-09-10 04:45:28.032199904 +0000 UTC m=+0.776039425 container remove 91597bdf09f78a2afbdfa498aa1f0ab74554b1e43940ff53c2f6a2020ac8687e (image=quay.ceph.io/ceph-ci/ceph:squid, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-mon-smithi112, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-09-10T04:45:28.299 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:45:28 smithi112 systemd[1]: ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@mon.smithi112.service: Deactivated successfully. 2024-09-10T04:45:28.299 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:45:28 smithi112 systemd[1]: Stopped Ceph mon.smithi112 for 513b3bc6-6f25-11ef-bcea-c7b262605968. 2024-09-10T04:45:28.299 INFO:journalctl@ceph.mon.smithi112.smithi112.stdout:Sep 10 04:45:28 smithi112 systemd[1]: ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@mon.smithi112.service: Consumed 28min 10.316s CPU time. 2024-09-10T04:45:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:28.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:28.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:28.862+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:29.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:28.863+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:29.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:28.743+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:29.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:28.743+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:29.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:28.743+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:29.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:28 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:28.743+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:29.564 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:29.209+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:29.564 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:29.209+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:29.564 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:29.209+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:29.565 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:29.210+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:29.830 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:29.755+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:29.755+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:29.755+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:29.831 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:29.756+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:30.130 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:29.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:30.131 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:29.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:30.131 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:29.866+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:30.131 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:29 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:29.867+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:30.428 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:30.222+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:30.429 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:30.222+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:30.429 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:30.222+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:30.429 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:30.223+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:31.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:30.886+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:30.886+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:30.886+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:31.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:30.887+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:31.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:30.789+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:31.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:30.789+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:31.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:30.789+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:31.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:30 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:30.789+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:31.420 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:31.193+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:31.420 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:31.193+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:31.420 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:31.193+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:31.421 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:31.193+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:31.911+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:31.911+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:32.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:31.911+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:32.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:31.911+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:32.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:31.808+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:32.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:31.808+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:32.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:31.808+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:32.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:31 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:31.809+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:32.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:32.205+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:32.205+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:32.205+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:32.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:32.206+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:32.902 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:32.878+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:32.902 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:32.878+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:32.902 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:32.878+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:32.903 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:32.879+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:32.903 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:32.802+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:32.903 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:32.802+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:32.903 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:32.802+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:32.903 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:32 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:32.803+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:33.450 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:33.173+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:33.451 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:33.173+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:33.451 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:33.173+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:33.451 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:33.175+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:33.873+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:33.873+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:33.873+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:34.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:33.874+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:34.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:33.813+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:34.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:33.813+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:34.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:33.813+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:34.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:33 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:33.814+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:34.380 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:34.215+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:34.381 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:34.215+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:34.381 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:34.215+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:34.381 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:34.215+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:34.929 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:34.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:34.929 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:34.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:34.929 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:34.840+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:34.929 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:34.840+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:34.930 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:34.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:34.930 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:34.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:34.930 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:34.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:34.930 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:34 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:34.783+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:35.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:35.177+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:35.177+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:35.177+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:35.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:35.178+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:36.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:35.818+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:35.818+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:35.818+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:36.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:35.818+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:36.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:35.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:36.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:35.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:36.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:35.782+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:36.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:35 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:35.783+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:36.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:36.202+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:36.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:36.202+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:36.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:36.202+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:36.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:36.203+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:36.843+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:36.843+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:36.843+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:37.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:36.843+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:37.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:36.820+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:37.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:36.820+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:37.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:36.820+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:37.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:36 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:36.821+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:37.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:37.250+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:37.250+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:37.250+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:37.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:37.251+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:37.981 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:37.879+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:37.981 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:37.879+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:37.982 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:37.879+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:37.982 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:37.880+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:37.982 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:37.835+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:37.982 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:37.835+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:37.982 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:37.835+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:37.982 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:37 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:37.836+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:38.580 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:38.283+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:38.283+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:38.283+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:38.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:38.284+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:38.870+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:38.870+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:38.870+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:39.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:38.871+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:39.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:38.802+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:39.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:38.802+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:39.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:38.802+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:39.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:38 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:38.803+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:39.521 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:39.298+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:39.521 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:39.298+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:39.522 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:39.298+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:39.522 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:39.298+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:40.071 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:39.852+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:40.071 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:39.852+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:40.072 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:39.852+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:40.072 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:39.853+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:40.072 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:39.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:40.072 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:39.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:40.072 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:39.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:40.072 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:39 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:39.808+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:40.330 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:40 smithi112 systemd[1]: Stopping Ceph osd.0 for 513b3bc6-6f25-11ef-bcea-c7b262605968... 2024-09-10T04:45:40.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:40.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:40.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:40.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:40.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:40.311+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:40.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:40.311+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:40.854 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:40.590+0000 7f9b64a80640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T04:45:40.854 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:40.590+0000 7f9b64a80640 -1 osd.0 46 *** Got signal Terminated *** 2024-09-10T04:45:40.854 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:40.590+0000 7f9b64a80640 -1 osd.0 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T04:45:40.855 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:40.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:40.855 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:40.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:40.855 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:40.835+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:40.855 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:40.836+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:41.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:40.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:40.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:40.852+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:41.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:40 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:40.853+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:41.812 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:41.331+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:41.813 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:41.332+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:41.813 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:41.332+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:41.813 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:41.332+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:42.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:41.811+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:42.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:41.811+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:42.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:41.811+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:42.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:41.811+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:42.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:41.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:42.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:41.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:42.083 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:41.827+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:42.084 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:41 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:41.828+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:42.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:42.359+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:42.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:42.359+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:42.776 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:42.359+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:42.777 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:42.360+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:43.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:42.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:42.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:42.774+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:43.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:42.774+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:42.779+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:43.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:42.779+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:43.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:42.779+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:43.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:42 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:42.780+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:43.740 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:43.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:43.741 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:43.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:43.741 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:43.410+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:43.741 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:43.410+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:44.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:43.752+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:43.752+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:43.752+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:44.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:43.753+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:44.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:43.739+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:44.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:43.739+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:44.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:43.739+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:44.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:43 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:43.740+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:44.765 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:44.426+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:44.766 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:44.426+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:44.766 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:44.426+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:44.766 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:44.426+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:45.080 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:44.764+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:44.764+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:44.764+0000 7f9b5d07d640 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:16.791550+0000 (oldest deadline 2024-09-10T04:35:36.791550+0000) 2024-09-10T04:45:45.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0[48774]: 2024-09-10T04:45:44.764+0000 7f9b5d07d640 -1 osd.0 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1118 2.19 2:9d0ed013:::200.0000001b:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:45.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:44.774+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:45.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:44.774+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:45.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:44.774+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:45.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:44 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:44.775+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:45.768 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:45.401+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:45.768 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:45.401+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:45.768 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:45.401+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:45.769 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:45.402+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:46.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:45.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:46.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:45.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:46.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:45.766+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:46.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:45 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:45.766+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:46.382 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:46 smithi112 podman[81394]: 2024-09-10 04:45:46.16928851 +0000 UTC m=+5.707865904 container died 274fcfdbe0d8eccc2a58254b4d59cabc85f7f8ff6f7e9be3a8ad2834925b43aa (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-10T04:45:46.633 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:46 smithi112 podman[81394]: 2024-09-10 04:45:46.617298951 +0000 UTC m=+6.155876355 container cleanup 274fcfdbe0d8eccc2a58254b4d59cabc85f7f8ff6f7e9be3a8ad2834925b43aa (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T04:45:46.634 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:46 smithi112 bash[81394]: ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0 2024-09-10T04:45:46.635 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:46.380+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:46.635 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:46.380+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:46.635 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:46.380+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:46.635 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:46.380+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:47.081 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:46 smithi112 podman[81414]: 2024-09-10 04:45:46.944380611 +0000 UTC m=+0.776689817 container remove 274fcfdbe0d8eccc2a58254b4d59cabc85f7f8ff6f7e9be3a8ad2834925b43aa (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-b53b6d3, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-10T04:45:47.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:46.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:47.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:46.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:47.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:46.754+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:47.082 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:46 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:46.755+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:47.580 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:47 smithi112 podman[81509]: 2024-09-10 04:45:47.215183597 +0000 UTC m=+0.017884175 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-10T04:45:47.581 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:47 smithi112 podman[81509]: 2024-09-10 04:45:47.337051184 +0000 UTC m=+0.139751761 container create ecba22e7f4268c3083d3aa9f2880c3abff8b20ce6657e886d32fb8699c2505a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3, io.buildah.version=1.37.2) 2024-09-10T04:45:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:47.331+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:47.331+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:47.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:47.331+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:47.582 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:47.332+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:47.718+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:47.718+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:47.718+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:48.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:47 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:47.718+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:48.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:47 smithi112 podman[81509]: 2024-09-10 04:45:47.614743903 +0000 UTC m=+0.417444480 container init ecba22e7f4268c3083d3aa9f2880c3abff8b20ce6657e886d32fb8699c2505a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, RELEASE=squid-b53b6d3, GIT_BRANCH=HEAD) 2024-09-10T04:45:48.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:47 smithi112 podman[81509]: 2024-09-10 04:45:47.618094559 +0000 UTC m=+0.420795143 container start ecba22e7f4268c3083d3aa9f2880c3abff8b20ce6657e886d32fb8699c2505a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0-deactivate, RELEASE=squid-b53b6d3, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-10T04:45:48.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:47 smithi112 podman[81509]: 2024-09-10 04:45:47.723347043 +0000 UTC m=+0.526047630 container attach ecba22e7f4268c3083d3aa9f2880c3abff8b20ce6657e886d32fb8699c2505a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3, ceph=True, CEPH_POINT_RELEASE=) 2024-09-10T04:45:48.082 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:47 smithi112 podman[81509]: 2024-09-10 04:45:47.816820505 +0000 UTC m=+0.619521099 container died ecba22e7f4268c3083d3aa9f2880c3abff8b20ce6657e886d32fb8699c2505a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-10T04:45:48.512 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:48.380+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:48.512 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:48.380+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:48.512 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:48.380+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:48.512 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:48.381+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:48 smithi112 podman[81534]: 2024-09-10 04:45:48.511295772 +0000 UTC m=+0.688531582 container remove ecba22e7f4268c3083d3aa9f2880c3abff8b20ce6657e886d32fb8699c2505a3 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-0-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-10T04:45:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:48 smithi112 systemd[1]: ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@osd.0.service: Deactivated successfully. 2024-09-10T04:45:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:48 smithi112 systemd[1]: Stopped Ceph osd.0 for 513b3bc6-6f25-11ef-bcea-c7b262605968. 2024-09-10T04:45:48.831 INFO:journalctl@ceph.osd.0.smithi112.stdout:Sep 10 04:45:48 smithi112 systemd[1]: ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@osd.0.service: Consumed 1h 40.062s CPU time. 2024-09-10T04:45:48.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:48.715+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:48.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:48.715+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:48.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:48.715+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:48.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:48 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:48.716+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:48.832 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:48 smithi112 systemd[1]: Stopping Ceph osd.1 for 513b3bc6-6f25-11ef-bcea-c7b262605968... 2024-09-10T04:45:49.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:49.039+0000 7f1daf169640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T04:45:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:49.039+0000 7f1daf169640 -1 osd.1 46 *** Got signal Terminated *** 2024-09-10T04:45:49.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:49.039+0000 7f1daf169640 -1 osd.1 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T04:45:49.745 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:49.333+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:49.745 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:49.333+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:49.745 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:49.333+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:49.745 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:49.334+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:50.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:49.742+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:50.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:49.742+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:50.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:49.742+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:50.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:49 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:49.743+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:50.787 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:50.383+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:50.787 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:50.383+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:50.787 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:50.383+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:50.787 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:50.384+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:51.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:50.784+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:51.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:50.784+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:51.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:50.784+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:51.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:50 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:50.785+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:51.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:51.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:51.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:51.375+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:51.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:51.376+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:52.330 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:51.833+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:51.833+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:51.833+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:52.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:51 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:51.833+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:52.816 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:52.390+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:52.816 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:52.390+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:52.817 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:52.390+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:52.817 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:52.391+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:53.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:52.813+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:52.813+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:52.813+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:53.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:52 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:52.814+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:53.809 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:53.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:53.809 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:53.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:53.809 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:53.404+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:53.809 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:53.404+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:54.080 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:53.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:53.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:53.807+0000 7f1da7766640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:37.165773+0000 (oldest deadline 2024-09-10T04:35:57.165773+0000) 2024-09-10T04:45:54.081 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:53 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1[56648]: 2024-09-10T04:45:53.808+0000 7f1da7766640 -1 osd.1 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.12:1116 3.4 3:27acd41b:::100000050d5.00000000:head [trimtrunc 2@3762197] snapc 1=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-09-10T04:45:54.642 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:54.373+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:54.643 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:54.373+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:54.643 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:54.373+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:54.643 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:54 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:54.373+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:54.942 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:54 smithi112 podman[81617]: 2024-09-10 04:45:54.641286115 +0000 UTC m=+5.725420229 container died c48f7eddf6a2509c944342ee145ed4bcc0563049a449749b507ef8be3dbb3931 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-10T04:45:55.326 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:55 smithi112 podman[81617]: 2024-09-10 04:45:55.040794689 +0000 UTC m=+6.124928803 container cleanup c48f7eddf6a2509c944342ee145ed4bcc0563049a449749b507ef8be3dbb3931 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-10T04:45:55.326 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:55 smithi112 bash[81617]: ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1 2024-09-10T04:45:55.580 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:55 smithi112 podman[81636]: 2024-09-10 04:45:55.325662149 +0000 UTC m=+0.680113660 container remove c48f7eddf6a2509c944342ee145ed4bcc0563049a449749b507ef8be3dbb3931 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1, org.label-schema.vendor=CentOS, RELEASE=squid-b53b6d3, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-10T04:45:55.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:55.358+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:55.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:55.358+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:55.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:55.358+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:55.581 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:55 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:55.358+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:55.972 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:55 smithi112 podman[81724]: 2024-09-10 04:45:55.563085183 +0000 UTC m=+0.017881625 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-10T04:45:55.972 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:55 smithi112 podman[81724]: 2024-09-10 04:45:55.710497777 +0000 UTC m=+0.165294202 container create 3f74c1077576452a95d785486d5c0029c1a45d6ca6bebb34b32141cb74eb7f58 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1-deactivate, RELEASE=squid-b53b6d3, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T04:45:55.972 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:55 smithi112 podman[81724]: 2024-09-10 04:45:55.971190365 +0000 UTC m=+0.425986793 container init 3f74c1077576452a95d785486d5c0029c1a45d6ca6bebb34b32141cb74eb7f58 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=squid-b53b6d3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True) 2024-09-10T04:45:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:55 smithi112 podman[81724]: 2024-09-10 04:45:55.974491068 +0000 UTC m=+0.429287498 container start 3f74c1077576452a95d785486d5c0029c1a45d6ca6bebb34b32141cb74eb7f58 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-b53b6d3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-09-10T04:45:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:56 smithi112 podman[81724]: 2024-09-10 04:45:56.062926799 +0000 UTC m=+0.517723230 container attach 3f74c1077576452a95d785486d5c0029c1a45d6ca6bebb34b32141cb74eb7f58 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=squid-b53b6d3, GIT_BRANCH=HEAD) 2024-09-10T04:45:56.331 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:56 smithi112 podman[81724]: 2024-09-10 04:45:56.183487676 +0000 UTC m=+0.638284104 container died 3f74c1077576452a95d785486d5c0029c1a45d6ca6bebb34b32141cb74eb7f58 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731) 2024-09-10T04:45:56.817 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:56.392+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:56.818 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:56.392+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:56.818 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:56.392+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:56.818 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:56 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:56.392+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:57.069 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:56 smithi112 podman[81747]: 2024-09-10 04:45:56.816738018 +0000 UTC m=+0.626946999 container remove 3f74c1077576452a95d785486d5c0029c1a45d6ca6bebb34b32141cb74eb7f58 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2) 2024-09-10T04:45:57.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:56 smithi112 systemd[1]: ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@osd.1.service: Deactivated successfully. 2024-09-10T04:45:57.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:56 smithi112 systemd[1]: Stopped Ceph osd.1 for 513b3bc6-6f25-11ef-bcea-c7b262605968. 2024-09-10T04:45:57.070 INFO:journalctl@ceph.osd.1.smithi112.stdout:Sep 10 04:45:56 smithi112 systemd[1]: ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@osd.1.service: Consumed 1h 53.962s CPU time. 2024-09-10T04:45:57.330 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:57 smithi112 systemd[1]: Stopping Ceph osd.2 for 513b3bc6-6f25-11ef-bcea-c7b262605968... 2024-09-10T04:45:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:57.361+0000 7fecb90a9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T04:45:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:57.361+0000 7fecb90a9640 -1 osd.2 46 *** Got signal Terminated *** 2024-09-10T04:45:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:57.361+0000 7fecb90a9640 -1 osd.2 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T04:45:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:57.364+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:57.364+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:57.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:57.364+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:57.832 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:57 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:57.365+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:58.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:58.408+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:58.408+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:58.408+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:58.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:58 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:58.409+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:45:59.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:59.389+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:59.389+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:59.389+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:45:59.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:45:59 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:45:59.390+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:46:00.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:46:00.355+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:46:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:46:00.355+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:46:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:46:00.355+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:46:00.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:00 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:46:00.356+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:46:01.830 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:46:01.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:46:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:46:01.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:46:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:46:01.367+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:46:01.831 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:01 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:46:01.368+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:46:02.807 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:46:02.340+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:46:02.807 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:46:02.340+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:46:02.807 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:46:02.340+0000 7fecb56ae640 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2024-09-10T04:35:28.960491+0000 (oldest deadline 2024-09-10T04:35:48.960491+0000) 2024-09-10T04:46:02.808 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:02 smithi112 ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2[64420]: 2024-09-10T04:46:02.341+0000 7fecb56ae640 -1 osd.2 46 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.24321.0:5169 3.1e 3:7ee72ee7:::10000004716.00000000:head [write 0~1333 in=1333b] snapc 1=[] ondisk+write+known_if_redirected e45) 2024-09-10T04:46:03.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:02 smithi112 podman[81830]: 2024-09-10 04:46:02.806541474 +0000 UTC m=+5.590322524 container died 2885646ffe21d9abb64a6a1344e0216dce69769563d1ff599da5dc5f48e532ec (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=squid-b53b6d3, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-10T04:46:03.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:03 smithi112 podman[81830]: 2024-09-10 04:46:03.178916456 +0000 UTC m=+5.962697508 container cleanup 2885646ffe21d9abb64a6a1344e0216dce69769563d1ff599da5dc5f48e532ec (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2, GIT_CLEAN=True, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-10T04:46:03.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:03 smithi112 bash[81830]: ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2 2024-09-10T04:46:03.747 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:03 smithi112 podman[81849]: 2024-09-10 04:46:03.438630918 +0000 UTC m=+0.628664139 container remove 2885646ffe21d9abb64a6a1344e0216dce69769563d1ff599da5dc5f48e532ec (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T04:46:04.025 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:03 smithi112 podman[81935]: 2024-09-10 04:46:03.649208683 +0000 UTC m=+0.015977610 image pull 54467abc55f864c4f73dfaf9c6ac3bb3e87b773123d5badaf790bee57321d594 quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66 2024-09-10T04:46:04.026 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:03 smithi112 podman[81935]: 2024-09-10 04:46:03.755952719 +0000 UTC m=+0.122721645 container create 6f3ba41fe48febc71f488f21dbb98f8c04c60b73cb7e9dc6d5a505f49549c066 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, ceph=True, RELEASE=squid-b53b6d3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-10T04:46:04.026 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:03 smithi112 podman[81935]: 2024-09-10 04:46:03.950077835 +0000 UTC m=+0.316846765 container init 6f3ba41fe48febc71f488f21dbb98f8c04c60b73cb7e9dc6d5a505f49549c066 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-b53b6d3, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-10T04:46:04.026 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:03 smithi112 podman[81935]: 2024-09-10 04:46:03.953482886 +0000 UTC m=+0.320251815 container start 6f3ba41fe48febc71f488f21dbb98f8c04c60b73cb7e9dc6d5a505f49549c066 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2-deactivate, RELEASE=squid-b53b6d3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-09-10T04:46:04.026 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:04 smithi112 podman[81935]: 2024-09-10 04:46:04.02477186 +0000 UTC m=+0.391540798 container attach 6f3ba41fe48febc71f488f21dbb98f8c04c60b73cb7e9dc6d5a505f49549c066 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=squid-b53b6d3) 2024-09-10T04:46:04.331 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:04 smithi112 podman[81935]: 2024-09-10 04:46:04.140271059 +0000 UTC m=+0.507039996 container died 6f3ba41fe48febc71f488f21dbb98f8c04c60b73cb7e9dc6d5a505f49549c066 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2-deactivate, RELEASE=squid-b53b6d3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-10T04:46:05.080 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:04 smithi112 podman[81959]: 2024-09-10 04:46:04.703393694 +0000 UTC m=+0.552913666 container remove 6f3ba41fe48febc71f488f21dbb98f8c04c60b73cb7e9dc6d5a505f49549c066 (image=quay.ceph.io/ceph-ci/ceph@sha256:3be30d720e37ec4df7f667e60f22b4d221c4ce7b408e68b90f9ac10220535e66, name=ceph-513b3bc6-6f25-11ef-bcea-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-b53b6d3, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-10T04:46:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:04 smithi112 systemd[1]: ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@osd.2.service: Deactivated successfully. 2024-09-10T04:46:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:04 smithi112 systemd[1]: Stopped Ceph osd.2 for 513b3bc6-6f25-11ef-bcea-c7b262605968. 2024-09-10T04:46:05.081 INFO:journalctl@ceph.osd.2.smithi112.stdout:Sep 10 04:46:04 smithi112 systemd[1]: ceph-513b3bc6-6f25-11ef-bcea-c7b262605968@osd.2.service: Consumed 31min 1.255s CPU time. 2024-09-10T04:46:08.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:46:08.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:46:11.229 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:46:20.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:46:20.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:46:23.297 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:46:35.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:46:35.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:46:38.369 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:46:41.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:46:41.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:46:41.441 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:46:41.441 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-10T04:46:41.441 INFO:tasks.cephadm:Teardown complete 2024-09-10T04:46:41.442 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephadm.py", line 1783, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephadm.py", line 2374, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephadm.py", line 2334, in task with contextutil.nested( File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephadm.py", line 1962, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephadm.py", line 232, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephadm.py", line 416, in _rm_cluster remote.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2024-09-10T04:46:41.469 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-10T04:46:41.483 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/install/__init__.py", line 641, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-09-10T04:46:41.499 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-10T04:46:41.499 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T04:46:41.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:46:41.516 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:46:44.573 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:46:53.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:46:53.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:46:56.637 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:47:08.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:47:08.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:47:11.709 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:47:14.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:47:14.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:47:14.781 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:47:14.781 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-10T04:47:14.828 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/install/__init__.py", line 641, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/install/__init__.py", line 637, in task with contextutil.nested( File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/install/__init__.py", line 221, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2024-09-10T04:47:14.850 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-10T04:47:14.860 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-10T04:47:14.861 DEBUG:teuthology.orchestra.run.smithi112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T04:47:14.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:47:14.870 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:47:14.886 INFO:teuthology.orchestra.run.smithi112.stderr:bash: line 1: ntpq: command not found 2024-09-10T04:47:14.890 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T04:47:14.890 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2024-09-10T04:47:14.890 INFO:teuthology.orchestra.run.smithi112.stdout:^* hv01.front.sepia.ceph.com 3 7 377 93 +13us[ -10us] +/- 43ms 2024-09-10T04:47:14.890 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 98 -299us[ -322us] +/- 50ms 2024-09-10T04:47:14.891 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 223 +461us[ +453us] +/- 78ms 2024-09-10T04:47:14.891 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T04:47:17.949 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:47:26.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:47:26.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:47:30.017 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:47:42.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:47:42.019 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:47:45.085 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:47:48.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:47:48.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:47:48.161 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:47:48.161 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-10T04:47:48.162 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2024-09-10T04:47:48.176 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-10T04:47:48.188 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-10T04:47:48.189 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-10T04:47:48.198 DEBUG:teuthology.orchestra.run.smithi112:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-10T04:47:48.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:47:48.285 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:47:51.357 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:48:00.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:48:00.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:48:03.425 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:48:15.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:48:15.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:48:18.493 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:48:21.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:48:21.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:48:21.565 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:48:21.565 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-10T04:48:21.565 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/selinux.py", line 166, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/selinux.py", line 196, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2024-09-10T04:48:21.572 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-10T04:48:21.584 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-10T04:48:21.594 INFO:teuthology.task.internal:Duration was 4919.969489 seconds 2024-09-10T04:48:21.594 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-10T04:48:21.603 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-10T04:48:21.603 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T04:48:21.609 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:48:21.609 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:48:21.653 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T04:48:24.673 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:48:33.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:48:33.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:48:36.737 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:48:48.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:48:48.739 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:48:51.805 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:48:54.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:48:54.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:48:54.877 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:48:54.877 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-10T04:48:54.878 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2024-09-10T04:48:54.894 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-10T04:48:54.907 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-10T04:48:54.908 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T04:48:54.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:48:54.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:48:58.013 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:49:07.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:49:07.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:49:10.077 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:49:22.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:49:22.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:49:25.149 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:49:28.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:49:28.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:49:28.221 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:49:28.221 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-10T04:49:28.222 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2024-09-10T04:49:28.236 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-10T04:49:28.248 DEBUG:teuthology.orchestra.run.smithi112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T04:49:28.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:49:28.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:49:28.281 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = core 2024-09-10T04:49:31.325 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:49:40.326 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:49:40.327 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:49:43.389 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:49:55.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:49:55.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:49:58.461 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:50:01.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:50:01.462 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:50:01.533 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:50:01.533 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-10T04:50:01.533 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2024-09-10T04:50:01.547 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-10T04:50:01.560 INFO:teuthology.task.internal:Transferring archived files... 2024-09-10T04:50:01.561 DEBUG:teuthology.misc:Transferring archived files from smithi112:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-09-10_02:59:23-orch:cephadm-wip-adk-testing-2024-09-09-1904-distro-default-smithi/7898573/remote/smithi112 2024-09-10T04:50:01.562 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T04:50:01.603 DEBUG:teuthology.misc:Transferring archived files from smithi169:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-09-10_02:59:23-orch:cephadm-wip-adk-testing-2024-09-09-1904-distro-default-smithi/7898573/remote/smithi169 2024-09-10T04:50:01.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:50:01.604 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:50:04.669 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:50:13.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:50:13.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:50:16.733 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:50:28.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:50:28.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:50:31.805 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:50:34.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:50:34.807 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:50:34.877 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:50:34.877 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-10T04:50:34.877 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/misc.py", line 735, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2024-09-10T04:50:34.894 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-10T04:50:34.911 INFO:teuthology.task.internal:Not uploading archives. 2024-09-10T04:50:34.911 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-10T04:50:34.927 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-10T04:50:34.927 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T04:50:34.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:50:34.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:50:34.951 INFO:teuthology.orchestra.run.smithi112.stdout: 262169 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 10 04:46 /home/ubuntu/cephtest 2024-09-10T04:50:34.952 INFO:teuthology.orchestra.run.smithi112.stdout: 265354 768 -rwxr-xr-x 1 ubuntu ubuntu 784498 Sep 10 03:32 /home/ubuntu/cephtest/cephadm 2024-09-10T04:50:34.952 INFO:teuthology.orchestra.run.smithi112.stdout: 262170 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Sep 10 04:49 /home/ubuntu/cephtest/archive 2024-09-10T04:50:34.952 INFO:teuthology.orchestra.run.smithi112.stdout: 262172 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 03:26 /home/ubuntu/cephtest/archive/syslog 2024-09-10T04:50:34.952 INFO:teuthology.orchestra.run.smithi112.stdout: 265324 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Sep 10 03:26 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T04:50:34.952 INFO:teuthology.orchestra.run.smithi112.stdout: 265325 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Sep 10 03:26 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T04:50:34.952 INFO:teuthology.orchestra.run.smithi112.stdout: 682225 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 04:47 /home/ubuntu/cephtest/archive/audit 2024-09-10T04:50:34.952 INFO:teuthology.orchestra.run.smithi112.stdout: 682227 184 -rw------- 1 ubuntu ubuntu 188161 Sep 10 04:47 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-09-10T04:50:34.953 INFO:teuthology.orchestra.run.smithi112.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-10T04:50:38.013 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:50:47.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:50:47.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:50:50.077 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:51:02.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:51:02.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:51:05.149 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:51:08.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-10T04:51:08.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T04:51:08.221 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-10T04:51:08.221 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (4) after waiting for 30 seconds 2024-09-10T04:51:08.221 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph-c_3a5e3f16720f432aea965c40a8f6eb480095ec6c/qa/tasks/cephfs/mount.py", line 219, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2024-09-10T04:51:08.237 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-10T04:51:08.250 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-10T04:51:08.262 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-09-10T04:51:08.266 INFO:teuthology.run:Summary data: description: orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 4919.969488620758 failure_reason: 'Command failed on smithi112 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 513b3bc6-6f25-11ef-bcea-c7b262605968 -e sha1=3a5e3f16720f432aea965c40a8f6eb480095ec6c -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | keys\''"\''"\'' | grep $sha1\''''' flavor: default owner: scheduled_adking@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3ebaa8e069d84b159680a2a04088c8c7 status: fail success: false 2024-09-10T04:51:08.267 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T04:51:08.349 INFO:teuthology.run:FAIL